Oct 31 00:30:31.652427 ip-10-0-143-188 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 00:30:31.652596 ip-10-0-143-188 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 00:30:31.652684 ip-10-0-143-188 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 00:30:31.653151 ip-10-0-143-188 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 00:30:41.780050 ip-10-0-143-188 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 00:30:41.780066 ip-10-0-143-188 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c53db8fc955b4ecc8c059a0a74ae7c16 -- Oct 31 00:32:43.146920 ip-10-0-143-188 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:32:43.594648 ip-10-0-143-188 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:32:43.594648 ip-10-0-143-188 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:32:43.594648 ip-10-0-143-188 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:32:43.594648 ip-10-0-143-188 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:32:43.600085 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.599994 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:32:43.600085 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600083 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:32:43.600085 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600087 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600091 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600095 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600097 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600100 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600103 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600105 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600108 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600111 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600113 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600116 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600119 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600121 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600124 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600126 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600129 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600131 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600135 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600144 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:32:43.600185 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600147 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600150 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600152 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600157 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600162 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600165 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600169 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600172 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600175 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600178 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600181 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600184 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600187 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600190 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600193 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600196 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600201 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600205 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600208 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:32:43.600661 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600211 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600214 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600217 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600220 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600223 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600225 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600227 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600230 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600232 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600235 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600238 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600241 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600243 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600246 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600248 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600252 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600255 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600258 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600268 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600272 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:32:43.601110 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600274 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600277 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600279 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600282 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600285 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600287 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600290 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600292 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600295 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600298 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600300 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600303 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600306 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600308 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600311 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600314 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600316 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600319 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600321 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600324 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:32:43.601584 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600327 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600330 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600332 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600335 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600338 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600340 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600342 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600345 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600348 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600351 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600354 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600356 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600359 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600361 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600363 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600366 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600368 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600371 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600374 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600376 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:32:43.602079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600379 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600381 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600383 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600386 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600388 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.600391 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601187 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601194 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601197 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601200 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601203 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601206 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601209 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601211 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601214 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601217 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601220 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601223 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601225 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:32:43.602546 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601228 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601231 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601234 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601236 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601239 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601242 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601244 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601247 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601249 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601252 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601254 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601257 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601259 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601262 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601264 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601267 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601269 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601272 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601274 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601279 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:32:43.603114 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601281 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601284 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601287 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601290 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601292 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601295 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601297 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601300 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601302 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601305 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601307 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601310 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601313 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601315 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601317 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601320 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601322 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601325 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601328 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:32:43.603577 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601330 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601333 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601335 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601339 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601341 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601344 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601346 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601349 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601352 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601354 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601357 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601359 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601362 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601365 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601367 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601370 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601373 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601376 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601378 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601381 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:32:43.604049 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601384 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601386 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601389 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601391 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601394 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601397 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601400 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601402 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601405 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601408 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601412 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601415 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601418 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601420 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601423 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601425 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601428 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601430 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:32:43.604520 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601433 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601437 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601442 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601445 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601447 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601450 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601452 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601455 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601459 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601461 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601464 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601466 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601469 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601471 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601474 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.601476 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602041 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602048 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602052 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:32:43.604958 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602055 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602059 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602062 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602065 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602068 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602072 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602075 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602078 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602081 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602084 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602088 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602090 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602093 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602096 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602098 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602101 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602104 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602106 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602109 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602112 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:32:43.605405 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602114 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602117 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602120 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602123 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602125 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602127 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602130 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602132 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602135 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602137 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602140 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602142 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602145 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602148 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602150 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602153 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602156 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602158 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602161 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602163 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:32:43.605886 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602167 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602170 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602174 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602177 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602179 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602182 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602185 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602189 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602192 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602194 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602197 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602199 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602202 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602205 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602208 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602211 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602214 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602216 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602219 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:32:43.606380 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602221 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602224 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602226 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602229 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602231 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602234 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602237 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602240 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602242 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602245 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602248 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602250 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602253 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602255 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602258 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602261 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602263 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602266 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602268 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602271 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:32:43.606851 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602274 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602276 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602279 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602282 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602284 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602287 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602289 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602292 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602295 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602300 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602304 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602306 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602309 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602311 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602314 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602317 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602319 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602322 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602324 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:32:43.607315 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602327 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602329 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602332 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602334 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.602337 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604113 2579 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604125 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604132 2579 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604137 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604142 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604145 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604150 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604154 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604157 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604160 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604164 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604167 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604171 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604174 2579 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604177 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604181 2579 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604184 2579 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604188 2579 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:32:43.607794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604193 2579 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604196 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604199 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604202 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604205 2579 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604209 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604212 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604215 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604219 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604222 2579 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604226 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604229 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604232 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604235 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604240 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604243 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604246 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604252 2579 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604255 2579 flags.go:64] FLAG: --enable-server="true" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604259 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604264 2579 flags.go:64] FLAG: --event-burst="100" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604267 2579 flags.go:64] FLAG: --event-qps="50" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604270 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604273 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:32:43.608343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604276 2579 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604287 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604291 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604294 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604297 2579 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604300 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604303 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604307 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604310 2579 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604313 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604316 2579 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604319 2579 flags.go:64] FLAG: --feature-gates="" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604323 2579 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604327 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604330 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604334 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604337 2579 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604340 2579 flags.go:64] FLAG: --help="false" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604343 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604347 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604350 2579 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604353 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604356 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:32:43.608960 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604360 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604363 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604367 2579 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604370 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604373 2579 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604376 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604379 2579 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604382 2579 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604385 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604388 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604391 2579 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604394 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604397 2579 flags.go:64] FLAG: --lock-file="" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604400 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604403 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604406 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604412 2579 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604415 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604419 2579 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604422 2579 flags.go:64] FLAG: --logging-format="text" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604425 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604429 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604432 2579 flags.go:64] FLAG: --manifest-url="" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604435 2579 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604440 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:32:43.609513 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604443 2579 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604447 2579 flags.go:64] FLAG: --max-pods="110" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604450 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604453 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604456 2579 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604459 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604462 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604465 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604468 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604476 2579 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604480 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604483 2579 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604486 2579 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604489 2579 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604492 2579 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604495 2579 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604498 2579 flags.go:64] FLAG: --port="10250" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604501 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604504 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0c0c4713797f47505" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604507 2579 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604510 2579 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604513 2579 flags.go:64] FLAG: --register-node="true" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604516 2579 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604521 2579 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604524 2579 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604527 2579 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:32:43.610109 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604531 2579 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604535 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604538 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604541 2579 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604545 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604548 2579 flags.go:64] FLAG: --runonce="false" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604551 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604554 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604557 2579 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604560 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604563 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604567 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604570 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604573 2579 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604576 2579 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604580 2579 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604583 2579 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604586 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604590 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604593 2579 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604596 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604602 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604605 2579 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604608 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604612 2579 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:32:43.610751 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604615 2579 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604618 2579 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604621 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604624 2579 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604628 2579 flags.go:64] FLAG: --v="2" Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604644 2579 flags.go:64] FLAG: --version="false" Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604649 2579 flags.go:64] FLAG: --vmodule="" Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604653 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.604657 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604798 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604802 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604805 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604807 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604810 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604813 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604815 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604818 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604821 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604823 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604826 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604828 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:32:43.611357 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604831 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604835 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604838 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604841 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604843 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604846 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604849 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604852 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604855 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604857 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604860 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604863 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604865 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604868 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604870 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604874 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604877 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604880 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604882 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:32:43.611884 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604885 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604888 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604890 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604893 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604896 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604898 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604901 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604903 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604906 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604908 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604911 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604914 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604918 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604922 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604926 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604929 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604932 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604935 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604938 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:32:43.612323 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604940 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604943 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604945 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604949 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604951 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604954 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604956 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604959 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604962 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604966 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604969 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604972 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604975 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604977 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604980 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604983 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604986 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604988 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604991 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604994 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:32:43.612876 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604996 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.604999 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605002 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605004 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605007 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605009 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605012 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605015 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605018 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605020 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605023 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605025 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605028 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605030 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605033 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605035 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605038 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605041 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605043 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605046 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:32:43.613356 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605048 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:32:43.613831 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605052 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:32:43.613831 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605054 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:32:43.613831 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605057 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:32:43.613831 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605059 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:32:43.613831 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605062 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:32:43.613831 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605064 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:32:43.613831 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605066 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:32:43.613831 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605069 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:32:43.613831 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605072 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:32:43.613831 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605074 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:32:43.613831 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605077 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:32:43.613831 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605081 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:32:43.613831 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605085 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:32:43.613831 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605088 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:32:43.613831 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605091 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:32:43.614179 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.605120 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:32:43.614179 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605156 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:32:43.614179 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605161 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:32:43.614179 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605164 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:32:43.614179 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605168 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:32:43.614179 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605171 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:32:43.614179 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605174 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:32:43.614179 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605176 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:32:43.614179 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605179 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:32:43.614179 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605182 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:32:43.614179 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605184 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:32:43.614179 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605187 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:32:43.614179 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605189 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:32:43.614179 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605192 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:32:43.614179 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605195 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:32:43.614179 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605197 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605200 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605203 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605206 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605208 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605211 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605214 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605217 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605219 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605222 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605225 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605228 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605230 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605233 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605235 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605239 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605243 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605246 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605249 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:32:43.614559 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605252 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605254 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605257 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605259 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605262 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605264 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605267 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605269 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605272 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605275 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605277 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605279 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605282 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605284 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605287 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605291 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605293 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605296 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605298 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605301 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:32:43.615028 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605303 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605307 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605310 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605312 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605315 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605318 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605320 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605323 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605325 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605328 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605331 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605333 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605336 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605338 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605341 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605343 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605346 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605348 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605351 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605353 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:32:43.615498 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605356 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605358 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605361 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605363 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605366 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605368 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605371 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605374 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605377 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605379 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605382 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605384 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605387 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605389 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605392 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605396 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605400 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605404 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605407 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:32:43.615988 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605410 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605413 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605417 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605420 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605422 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605425 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605427 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605430 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605433 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605435 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605438 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605440 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605443 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.605445 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.612458 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.612476 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.612495 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.612499 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612563 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:32:43.616432 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612568 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612571 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612574 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612577 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612580 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612582 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612585 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612588 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612591 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612594 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612597 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612599 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612602 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612605 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612607 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612610 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612613 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612615 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612618 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:32:43.616961 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612621 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612624 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612627 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612630 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612648 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612652 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612655 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612658 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612661 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612665 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612668 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612671 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612674 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612676 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612679 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612682 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612685 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612687 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612692 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:32:43.617406 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612696 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612700 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612703 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612706 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612708 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612711 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612714 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612716 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612719 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612721 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612724 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612726 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612729 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612732 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612735 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612738 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612740 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612743 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612746 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612749 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:32:43.618738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612752 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612754 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612757 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612760 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612763 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612766 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612769 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612771 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612774 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612776 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612779 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612781 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612784 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612786 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612789 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612792 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612795 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612797 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612799 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612802 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:32:43.619215 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612804 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612807 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612810 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612813 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612815 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612818 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612821 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612824 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612827 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612829 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612832 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612835 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612837 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612840 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612843 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612846 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612850 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612853 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612856 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612859 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:32:43.619692 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612861 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:32:43.620180 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612864 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:32:43.620180 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612866 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:32:43.620180 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612869 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:32:43.620180 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612872 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:32:43.620180 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612874 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:32:43.620180 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.612877 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:32:43.620180 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.612905 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:32:43.620180 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613070 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:32:43.620180 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613074 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:32:43.620180 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613078 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:32:43.620180 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613081 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:32:43.620180 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613084 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:32:43.620180 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613086 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:32:43.620180 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613090 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:32:43.620180 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613092 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:32:43.620180 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613095 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613098 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613101 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613103 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613106 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613108 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613111 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613113 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613115 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613118 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613120 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613123 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613125 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613129 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613131 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613134 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613136 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613139 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613142 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:32:43.620561 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613144 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613146 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613149 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613151 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613154 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613157 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613159 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613161 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613164 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613166 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613169 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613172 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613174 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613177 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613179 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613182 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613185 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613187 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613190 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613192 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:32:43.621058 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613194 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613197 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613199 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613202 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613204 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613207 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613209 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613213 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613217 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613220 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613223 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613226 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613230 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613233 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613237 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613240 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613242 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613245 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613247 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613250 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:32:43.621894 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613252 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613255 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613257 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613260 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613263 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613265 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613268 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613270 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613273 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613275 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613278 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613280 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613283 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613285 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613287 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613290 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613292 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613295 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613297 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:32:43.622618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613300 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613303 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613305 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613308 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613310 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613313 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613315 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613318 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613322 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613326 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613328 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613331 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613334 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613337 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613339 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613342 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613344 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613347 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613349 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:32:43.623133 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613353 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:32:43.623618 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.613375 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:32:43.623618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613405 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:32:43.623618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613409 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:32:43.623618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613412 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:32:43.623618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613415 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:32:43.623618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613417 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:32:43.623618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613420 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:32:43.623618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613422 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:32:43.623618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613425 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:32:43.623618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613427 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:32:43.623618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613430 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:32:43.623618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613433 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:32:43.623618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613435 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:32:43.623618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613438 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:32:43.623618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613441 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:32:43.623618 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613443 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613447 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613449 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613451 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613454 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613457 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613459 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613462 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613464 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613467 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613469 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613472 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613474 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613477 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613479 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613482 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613485 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613487 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:32:43.624134 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613491 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613494 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613497 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613500 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613503 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613505 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613508 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613510 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613512 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613515 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613517 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613520 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613522 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613525 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613528 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613530 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613533 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613535 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613538 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613543 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:32:43.624557 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613546 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613549 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613551 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613554 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613556 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613559 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613561 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613564 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613567 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613569 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613572 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613575 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613578 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613581 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613583 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613586 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613588 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613591 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613594 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:32:43.625143 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613596 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613599 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613601 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613604 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613607 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613610 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613612 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613615 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613618 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613621 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613623 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613626 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613630 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613647 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613650 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613654 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613656 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613659 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613662 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613664 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:32:43.625589 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613667 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613670 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613672 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613675 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613677 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613680 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613682 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613686 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613689 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613693 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613696 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613698 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613702 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613705 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:43.613707 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.613763 2579 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.617064 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.618208 2579 server.go:1014] "Starting client certificate rotation" Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.618715 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:32:43.626101 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.618752 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:32:43.626605 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.621999 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:32:43.626605 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.626265 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:32:43.640770 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.640747 2579 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:32:43.644768 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.644748 2579 log.go:25] "Validated CRI v1 image API" Oct 31 00:32:43.645591 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.645538 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:32:43.648425 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.648402 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 862cc37e-30b9-42ec-831f-b1b4f8ed4b93:/dev/nvme0n1p3 8a805194-6aa5-477c-a701-efd66da57918:/dev/nvme0n1p4] Oct 31 00:32:43.648492 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.648424 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 00:32:43.653785 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.653657 2579 manager.go:217] Machine: {Timestamp:2026-10-31 00:32:43.652194685 +0000 UTC m=+0.373509684 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100021 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec230cfd53d2a7adfdada89d18b0a3d4 SystemUUID:ec230cfd-53d2-a7ad-fdad-a89d18b0a3d4 BootID:c53db8fc-955b-4ecc-8c05-9a0a74ae7c16 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f8:3e:94:99 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f8:3e:94:99 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:d4:0e:89:d6:74 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:32:43.653785 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.653778 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:32:43.653920 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.653756 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:32:43.653920 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.653869 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:32:43.657425 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.657396 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:32:43.657565 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.657427 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-188.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:32:43.657613 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.657567 2579 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:32:43.657613 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.657576 2579 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:32:43.657613 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.657610 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:32:43.658266 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.658255 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:32:43.659509 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.659498 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:32:43.664294 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.664282 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:32:43.664413 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.664403 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:32:43.664447 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.664439 2579 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:32:43.664478 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.664450 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:32:43.664478 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.664461 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:32:43.664478 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.664471 2579 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:32:43.664576 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.664480 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:32:43.664827 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.664816 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:32:43.668340 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.668312 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:32:43.669928 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.669911 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5nbhd" Oct 31 00:32:43.676291 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.676274 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:32:43.677146 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.677133 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:32:43.677197 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.677164 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:32:43.677347 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.677335 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:32:43.677387 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.677357 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:32:43.677387 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.677367 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:32:43.677387 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.677377 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:32:43.677387 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.677385 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:32:43.677541 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.677395 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:32:43.677541 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.677457 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5nbhd" Oct 31 00:32:43.678164 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.678153 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:32:43.678195 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.678166 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:32:43.678195 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.678173 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:32:43.678195 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.678180 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:32:43.678195 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.678189 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:32:43.678327 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.678198 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:32:43.679898 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.679878 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:32:43.679898 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.679895 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:32:43.684287 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.684272 2579 server.go:1263] "Started kubelet" Oct 31 00:32:43.684364 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.684322 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:32:43.684434 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.684391 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:32:43.684483 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.684455 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:32:43.685145 ip-10-0-143-188 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:32:43.686247 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.686229 2579 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:32:43.686578 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.686564 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:32:43.687757 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.687740 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:32:43.689460 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.689445 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-143-188.us-east-2.compute.internal" not found Oct 31 00:32:43.690072 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.690052 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:32:43.691190 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:43.691158 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 00:32:43.692664 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.692645 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:32:43.693568 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.693547 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:32:43.693708 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:43.693674 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-188.us-east-2.compute.internal\" not found" Oct 31 00:32:43.694397 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.694379 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:32:43.694397 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.694386 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:32:43.694503 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.694405 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:32:43.694544 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.694521 2579 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:32:43.694544 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.694529 2579 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:32:43.695424 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.695410 2579 factory.go:55] Registering systemd factory Oct 31 00:32:43.695424 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.695425 2579 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:32:43.695661 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.695647 2579 factory.go:153] Registering CRI-O factory Oct 31 00:32:43.695724 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.695665 2579 factory.go:223] Registration of the crio container factory successfully Oct 31 00:32:43.695724 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.695710 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:32:43.695724 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.695724 2579 factory.go:103] Registering Raw factory Oct 31 00:32:43.695852 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.695734 2579 manager.go:1199] Started watching for new ooms in manager Oct 31 00:32:43.697094 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.697075 2579 manager.go:319] Starting recovery of all containers Oct 31 00:32:43.699462 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.699444 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:32:43.702204 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:43.702182 2579 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-188.us-east-2.compute.internal\" not found" node="ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:43.702861 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.702839 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-143-188.us-east-2.compute.internal" not found Oct 31 00:32:43.707946 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.707929 2579 manager.go:324] Recovery completed Oct 31 00:32:43.713227 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.713212 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:32:43.715378 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.715360 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:32:43.715454 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.715392 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:32:43.715454 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.715402 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:32:43.716054 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.716041 2579 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:32:43.716054 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.716053 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:32:43.716151 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.716070 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:32:43.716151 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.716045 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:32:43.716801 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.716785 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:32:43.716860 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.716812 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:32:43.716860 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.716824 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:32:43.760516 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.760488 2579 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-143-188.us-east-2.compute.internal" not found Oct 31 00:32:43.793805 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:43.793772 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-188.us-east-2.compute.internal\" not found" Oct 31 00:32:43.855230 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.855166 2579 policy_none.go:50] "Start" Oct 31 00:32:43.855230 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.855196 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:32:43.855230 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.855223 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:32:43.857299 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.857289 2579 policy_none.go:44] "Start" Oct 31 00:32:43.882080 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.879343 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:32:43.882080 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.880708 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:32:43.882080 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.880731 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:32:43.882080 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.880753 2579 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:32:43.882080 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:43.880792 2579 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 00:32:43.884915 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.884898 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:32:43.886230 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.886205 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:32:43.888490 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.888474 2579 manager.go:341] "Starting Device Plugin manager" Oct 31 00:32:43.888588 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:43.888501 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:32:43.888588 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.888512 2579 server.go:87] "Starting device plugin registration server" Oct 31 00:32:43.888794 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.888780 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:32:43.888842 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.888796 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:32:43.888884 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.888867 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:32:43.888969 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.888941 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:32:43.888969 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.888954 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:32:43.889454 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:43.889426 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:32:43.889519 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:43.889479 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-188.us-east-2.compute.internal\" not found" Oct 31 00:32:43.981273 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.981237 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal"] Oct 31 00:32:43.981455 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.981315 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:32:43.982308 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.982292 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:32:43.982390 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.982323 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:32:43.982390 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.982336 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:32:43.984699 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.984687 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:32:43.984857 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.984840 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:43.984919 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.984878 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:32:43.985421 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.985400 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:32:43.985517 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.985430 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:32:43.985517 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.985441 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:32:43.985517 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.985405 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:32:43.985689 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.985520 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:32:43.985689 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.985537 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:32:43.987679 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.987664 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:43.987763 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.987693 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:32:43.988395 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.988377 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:32:43.988478 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.988404 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:32:43.988478 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.988426 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:32:43.989675 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.989661 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:32:43.990360 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.990341 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:32:43.990448 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.990370 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:32:43.990448 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.990380 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:32:43.990448 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.990400 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:43.999658 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:43.999614 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:43.999743 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:43.999657 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-188.us-east-2.compute.internal\": node \"ip-10-0-143-188.us-east-2.compute.internal\" not found" Oct 31 00:32:44.011548 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.011525 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:44.014083 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.014069 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:44.028177 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.028151 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:32:44.028177 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.028177 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:32:44.094533 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.094493 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:44.095747 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.095726 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72fb96218f1d813dd9872d369c087ed5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"72fb96218f1d813dd9872d369c087ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:44.095822 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.095756 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72fb96218f1d813dd9872d369c087ed5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"72fb96218f1d813dd9872d369c087ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:44.095822 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.095796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c7fa48eaf6754635db0f21b009865b16-config\") pod \"kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"c7fa48eaf6754635db0f21b009865b16\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:44.114205 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.114158 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:32:44.114205 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:44.114197 2579 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:44.114283 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.114209 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:44.132760 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.132733 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:32:44.132887 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:44.132771 2579 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:44.196463 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.196429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72fb96218f1d813dd9872d369c087ed5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"72fb96218f1d813dd9872d369c087ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:44.196463 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.196466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72fb96218f1d813dd9872d369c087ed5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"72fb96218f1d813dd9872d369c087ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:44.196702 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.196491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c7fa48eaf6754635db0f21b009865b16-config\") pod \"kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"c7fa48eaf6754635db0f21b009865b16\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:44.196702 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.196548 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c7fa48eaf6754635db0f21b009865b16-config\") pod \"kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"c7fa48eaf6754635db0f21b009865b16\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:44.196702 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.196551 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72fb96218f1d813dd9872d369c087ed5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"72fb96218f1d813dd9872d369c087ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:44.196702 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.196550 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72fb96218f1d813dd9872d369c087ed5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"72fb96218f1d813dd9872d369c087ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:44.334818 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.334792 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:44.335995 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.335976 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:32:44.618078 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.618055 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 00:32:44.618675 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.618220 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:32:44.618675 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.618220 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:32:44.618675 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.618240 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:32:44.666541 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.666498 2579 apiserver.go:52] "Watching apiserver" Oct 31 00:32:44.674864 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.674832 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:32:44.675210 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.675188 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-czkrf","openshift-multus/multus-6mcgm","openshift-network-operator/iptables-alerter-ldwwp","openshift-ovn-kubernetes/ovnkube-node-h5226","kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-2662j","openshift-network-diagnostics/network-check-target-6ftrv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726","openshift-dns/node-resolver-9fz4x","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal","kube-system/global-pull-secret-syncer-7tgpr","openshift-image-registry/node-ca-n58v5","openshift-multus/multus-additional-cni-plugins-hkql2","kube-system/konnectivity-agent-65kvb"] Oct 31 00:32:44.677863 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.677841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.680106 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.680088 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.681895 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.681870 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:32:44.681980 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.681873 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:32:44.681980 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.681915 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jwzf5\"" Oct 31 00:32:44.682355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.682342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:32:44.682620 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.682573 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:27:44 +0000 UTC" deadline="2028-06-01 22:01:10.775536692 +0000 UTC" Oct 31 00:32:44.682620 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.682620 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13917h28m26.092920013s" Oct 31 00:32:44.683393 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.683375 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:32:44.683393 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.683387 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:32:44.683807 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.683785 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k5rkn\"" Oct 31 00:32:44.683887 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.683812 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:32:44.686416 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.683949 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:32:44.686416 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.685283 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:32:44.686416 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.685439 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.686416 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.685672 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:32:44.686416 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.685832 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cvzdq\"" Oct 31 00:32:44.686416 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.686086 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:32:44.687435 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.687418 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:44.687504 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:44.687486 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2662j" podUID="0d3524d6-1b11-41e9-9862-802d687ea87b" Oct 31 00:32:44.689191 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.689178 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:32:44.689752 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.689735 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:32:44.690443 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:44.690426 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ftrv" podUID="51aeac6f-2315-42d4-b63e-47efd057cf91" Oct 31 00:32:44.692029 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.692017 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.693409 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.693394 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 00:32:44.694387 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.694371 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:32:44.696813 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.696796 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:44.696913 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:44.696890 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tgpr" podUID="ffd93b08-b857-4571-a1b5-5eaca4456f8b" Oct 31 00:32:44.699247 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699230 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:32:44.699345 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699297 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-kubelet\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.699345 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699333 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-slash\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.699463 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovnkube-config\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.699463 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-host\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.699463 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699411 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:44.699463 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699438 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-var-lib-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.699673 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699498 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-cni-netd\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.699673 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovnkube-script-lib\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.699673 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwqpz\" (UniqueName: \"kubernetes.io/projected/993fd395-dba5-4c79-99d3-72792b8c273c-kube-api-access-zwqpz\") pod \"node-resolver-9fz4x\" (UID: \"993fd395-dba5-4c79-99d3-72792b8c273c\") " pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:32:44.699673 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699623 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-k8s-cni-cncf-io\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.699673 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699645 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9pf2\" (UniqueName: \"kubernetes.io/projected/0d3524d6-1b11-41e9-9862-802d687ea87b-kube-api-access-q9pf2\") pod \"network-metrics-daemon-2662j\" (UID: \"0d3524d6-1b11-41e9-9862-802d687ea87b\") " pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:44.699909 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.699909 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-run\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.699909 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699768 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-multus-certs\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.699909 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovn-node-metrics-cert\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.699909 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699899 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/993fd395-dba5-4c79-99d3-72792b8c273c-tmp-dir\") pod \"node-resolver-9fz4x\" (UID: \"993fd395-dba5-4c79-99d3-72792b8c273c\") " pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:32:44.700074 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-sys\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.700074 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.699995 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-system-cni-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.700074 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700036 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-cni-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.700074 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700064 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.700265 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700089 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-registration-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.700265 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700136 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de6e63f3-65e8-4d4d-bad4-015fed1391a3-cni-binary-copy\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.700265 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700181 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-cni-multus\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.700265 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700218 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.700560 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700291 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysctl-conf\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.700560 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-var-lib-kubelet\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.700560 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700384 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs\") pod \"network-metrics-daemon-2662j\" (UID: \"0d3524d6-1b11-41e9-9862-802d687ea87b\") " pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:44.700560 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700452 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn99g\" (UniqueName: \"kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g\") pod \"network-check-target-6ftrv\" (UID: \"51aeac6f-2315-42d4-b63e-47efd057cf91\") " pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:32:44.700560 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.700560 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-device-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.700804 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-sys-fs\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.700804 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-systemd\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.700804 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700629 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-kubelet\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.700804 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-etc-kubernetes\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.700804 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700746 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-cni-bin\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.700971 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700802 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysconfig\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.700971 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-modprobe-d\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.700971 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/560532f2-4a95-4180-a772-b269efd3d73d-etc-tuned\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.700971 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/560532f2-4a95-4180-a772-b269efd3d73d-tmp\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.700971 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700924 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfw88\" (UniqueName: \"kubernetes.io/projected/de6e63f3-65e8-4d4d-bad4-015fed1391a3-kube-api-access-qfw88\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.700971 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.700956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffd93b08-b857-4571-a1b5-5eaca4456f8b-kubelet-config\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:44.701150 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-run-netns\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.701150 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-log-socket\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.701150 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gvtc\" (UniqueName: \"kubernetes.io/projected/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-kube-api-access-8gvtc\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.701150 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-kubernetes\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.701150 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701115 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-socket-dir-parent\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.701150 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-host-slash\") pod \"iptables-alerter-ldwwp\" (UID: \"64dd929e-8d9d-47f3-bcb9-a0371d0fc279\") " pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:32:44.701359 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701179 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-ovn\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.701359 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/993fd395-dba5-4c79-99d3-72792b8c273c-hosts-file\") pod \"node-resolver-9fz4x\" (UID: \"993fd395-dba5-4c79-99d3-72792b8c273c\") " pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:32:44.701359 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9tcp\" (UniqueName: \"kubernetes.io/projected/560532f2-4a95-4180-a772-b269efd3d73d-kube-api-access-h9tcp\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.701359 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-cnibin\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.701359 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-hostroot\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.701544 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffd93b08-b857-4571-a1b5-5eaca4456f8b-dbus\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:44.701544 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-etc-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.701544 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701410 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h6qd\" (UniqueName: \"kubernetes.io/projected/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-kube-api-access-2h6qd\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.701544 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701439 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-netns\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.701544 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701458 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-env-overrides\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.701544 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701486 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-etc-selinux\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.701544 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-lib-modules\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.701852 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-cni-bin\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.701852 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701628 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-iptables-alerter-script\") pod \"iptables-alerter-ldwwp\" (UID: \"64dd929e-8d9d-47f3-bcb9-a0371d0fc279\") " pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:32:44.701852 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-node-log\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.701852 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701721 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-socket-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.701852 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysctl-d\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.701852 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701818 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpnf2\" (UniqueName: \"kubernetes.io/projected/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-kube-api-access-xpnf2\") pod \"iptables-alerter-ldwwp\" (UID: \"64dd929e-8d9d-47f3-bcb9-a0371d0fc279\") " pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:32:44.701852 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701850 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-systemd-units\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.702078 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701876 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-systemd\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.702078 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701918 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-conf-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.702078 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-daemon-config\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.702078 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.701948 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.704246 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.704230 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-65kvb" Oct 31 00:32:44.730683 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.730655 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:32:44.730796 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.730747 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vjmgp" Oct 31 00:32:44.740479 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.740453 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vjmgp" Oct 31 00:32:44.790459 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.790429 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:32:44.790459 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.790426 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:32:44.790459 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.790457 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:32:44.790692 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.790498 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:32:44.790880 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.790859 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hkzgb\"" Oct 31 00:32:44.790880 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.790868 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:32:44.791875 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.791860 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:32:44.791927 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.791893 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ppkcp\"" Oct 31 00:32:44.792052 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.792039 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:32:44.792300 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.792284 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:32:44.792357 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.792309 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:32:44.792357 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.792318 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:32:44.792357 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.792321 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:32:44.792357 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.792354 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:32:44.792563 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.792362 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:32:44.792563 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.792366 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kxg2f\"" Oct 31 00:32:44.792563 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.792383 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:32:44.792563 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.792395 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:32:44.792563 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.792402 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:32:44.792563 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.792414 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b29sj\"" Oct 31 00:32:44.792563 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.792418 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8fmps\"" Oct 31 00:32:44.792563 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.792428 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:32:44.792893 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.792880 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lfknl\"" Oct 31 00:32:44.795551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.795534 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:32:44.802194 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffd93b08-b857-4571-a1b5-5eaca4456f8b-kubelet-config\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:44.802287 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-run-netns\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.802287 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-log-socket\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.802287 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802247 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvtc\" (UniqueName: \"kubernetes.io/projected/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-kube-api-access-8gvtc\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.802404 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802306 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-log-socket\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.802404 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802305 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-run-netns\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.802404 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802307 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffd93b08-b857-4571-a1b5-5eaca4456f8b-kubelet-config\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:44.802404 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-kubernetes\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.802404 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802365 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-kubernetes\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.802404 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-socket-dir-parent\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.802585 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-host-slash\") pod \"iptables-alerter-ldwwp\" (UID: \"64dd929e-8d9d-47f3-bcb9-a0371d0fc279\") " pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:32:44.802585 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802438 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-host-slash\") pod \"iptables-alerter-ldwwp\" (UID: \"64dd929e-8d9d-47f3-bcb9-a0371d0fc279\") " pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:32:44.802585 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802448 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3346ce41-dd23-4e63-8944-43f2d7d4aaec-konnectivity-ca\") pod \"konnectivity-agent-65kvb\" (UID: \"3346ce41-dd23-4e63-8944-43f2d7d4aaec\") " pod="kube-system/konnectivity-agent-65kvb" Oct 31 00:32:44.802585 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802470 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-socket-dir-parent\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.802585 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802480 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-ovn\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.802585 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802521 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/993fd395-dba5-4c79-99d3-72792b8c273c-hosts-file\") pod \"node-resolver-9fz4x\" (UID: \"993fd395-dba5-4c79-99d3-72792b8c273c\") " pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:32:44.802585 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802542 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-ovn\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.802585 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802542 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9tcp\" (UniqueName: \"kubernetes.io/projected/560532f2-4a95-4180-a772-b269efd3d73d-kube-api-access-h9tcp\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.802585 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802574 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-cnibin\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.802585 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802584 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/993fd395-dba5-4c79-99d3-72792b8c273c-hosts-file\") pod \"node-resolver-9fz4x\" (UID: \"993fd395-dba5-4c79-99d3-72792b8c273c\") " pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:32:44.802981 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-hostroot\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.802981 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802616 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-cnibin\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.802981 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802617 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-system-cni-dir\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.802981 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3346ce41-dd23-4e63-8944-43f2d7d4aaec-agent-certs\") pod \"konnectivity-agent-65kvb\" (UID: \"3346ce41-dd23-4e63-8944-43f2d7d4aaec\") " pod="kube-system/konnectivity-agent-65kvb" Oct 31 00:32:44.802981 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffd93b08-b857-4571-a1b5-5eaca4456f8b-dbus\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:44.802981 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802727 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffd93b08-b857-4571-a1b5-5eaca4456f8b-dbus\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:44.802981 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-etc-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.802981 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802766 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h6qd\" (UniqueName: \"kubernetes.io/projected/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-kube-api-access-2h6qd\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.802981 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802639 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-hostroot\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.802981 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-netns\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.802981 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802793 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-etc-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.802981 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91f27c45-482c-4a05-998e-18b57ebfa499-serviceca\") pod \"node-ca-n58v5\" (UID: \"91f27c45-482c-4a05-998e-18b57ebfa499\") " pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:32:44.802981 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-env-overrides\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.802981 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802856 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-netns\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.802981 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-etc-selinux\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.803725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.802991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-lib-modules\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.803725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803004 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-etc-selinux\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.803725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-cni-bin\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.803725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-iptables-alerter-script\") pod \"iptables-alerter-ldwwp\" (UID: \"64dd929e-8d9d-47f3-bcb9-a0371d0fc279\") " pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:32:44.803725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803067 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-node-log\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.803725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803090 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-cni-bin\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.803725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803094 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-socket-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.803725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803103 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-lib-modules\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.803725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysctl-d\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.803725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803163 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-node-log\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.803725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpnf2\" (UniqueName: \"kubernetes.io/projected/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-kube-api-access-xpnf2\") pod \"iptables-alerter-ldwwp\" (UID: \"64dd929e-8d9d-47f3-bcb9-a0371d0fc279\") " pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:32:44.803725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803204 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysctl-d\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.803725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803213 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.803725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803251 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.803725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-systemd-units\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.803725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-systemd\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.803725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-conf-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803341 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-systemd-units\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803344 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-conf-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803361 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-systemd\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-daemon-config\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-kubelet\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803416 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-slash\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovnkube-config\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803454 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-host\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803480 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803486 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-slash\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803489 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-kubelet\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-var-lib-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-cni-netd\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803576 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-var-lib-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovnkube-script-lib\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803616 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwqpz\" (UniqueName: \"kubernetes.io/projected/993fd395-dba5-4c79-99d3-72792b8c273c-kube-api-access-zwqpz\") pod \"node-resolver-9fz4x\" (UID: \"993fd395-dba5-4c79-99d3-72792b8c273c\") " pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:44.803581 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:32:44.804551 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-k8s-cni-cncf-io\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.805547 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803702 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-host\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.805547 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803703 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-k8s-cni-cncf-io\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.805547 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803618 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-cni-netd\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.805547 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:44.803720 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret podName:ffd93b08-b857-4571-a1b5-5eaca4456f8b nodeName:}" failed. No retries permitted until 2026-10-31 00:32:45.303667791 +0000 UTC m=+2.024982799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret") pod "global-pull-secret-syncer-7tgpr" (UID: "ffd93b08-b857-4571-a1b5-5eaca4456f8b") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:32:44.805547 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803806 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9pf2\" (UniqueName: \"kubernetes.io/projected/0d3524d6-1b11-41e9-9862-802d687ea87b-kube-api-access-q9pf2\") pod \"network-metrics-daemon-2662j\" (UID: \"0d3524d6-1b11-41e9-9862-802d687ea87b\") " pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:44.805547 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803835 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91f27c45-482c-4a05-998e-18b57ebfa499-host\") pod \"node-ca-n58v5\" (UID: \"91f27c45-482c-4a05-998e-18b57ebfa499\") " pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:32:44.805547 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.805547 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803875 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-socket-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.805547 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-run\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.805547 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-multus-certs\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.805547 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803940 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-run\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.805547 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803957 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n24w\" (UniqueName: \"kubernetes.io/projected/21754645-32d8-4b1b-a7f5-00594f5854f3-kube-api-access-6n24w\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.805547 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803983 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-multus-certs\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.805547 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803985 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.805547 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.803995 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovn-node-metrics-cert\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.805547 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/993fd395-dba5-4c79-99d3-72792b8c273c-tmp-dir\") pod \"node-resolver-9fz4x\" (UID: \"993fd395-dba5-4c79-99d3-72792b8c273c\") " pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-sys\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-system-cni-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804164 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-cni-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804217 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-registration-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de6e63f3-65e8-4d4d-bad4-015fed1391a3-cni-binary-copy\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804276 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-env-overrides\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804297 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-cni-multus\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804324 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rblj9\" (UniqueName: \"kubernetes.io/projected/91f27c45-482c-4a05-998e-18b57ebfa499-kube-api-access-rblj9\") pod \"node-ca-n58v5\" (UID: \"91f27c45-482c-4a05-998e-18b57ebfa499\") " pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804385 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-sys\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804420 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/993fd395-dba5-4c79-99d3-72792b8c273c-tmp-dir\") pod \"node-resolver-9fz4x\" (UID: \"993fd395-dba5-4c79-99d3-72792b8c273c\") " pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804432 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804473 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-cni-multus\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804484 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-cni-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804483 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovnkube-script-lib\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804506 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-daemon-config\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.806368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804499 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-system-cni-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804521 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-registration-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysctl-conf\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804597 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-var-lib-kubelet\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs\") pod \"network-metrics-daemon-2662j\" (UID: \"0d3524d6-1b11-41e9-9862-802d687ea87b\") " pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804736 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysctl-conf\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn99g\" (UniqueName: \"kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g\") pod \"network-check-target-6ftrv\" (UID: \"51aeac6f-2315-42d4-b63e-47efd057cf91\") " pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804767 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-var-lib-kubelet\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-device-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804832 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804833 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-sys-fs\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804893 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-device-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:44.804835 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804895 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.807355 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804916 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovnkube-config\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.808256 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:44.804959 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs podName:0d3524d6-1b11-41e9-9862-802d687ea87b nodeName:}" failed. No retries permitted until 2026-10-31 00:32:45.304942054 +0000 UTC m=+2.026257059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs") pod "network-metrics-daemon-2662j" (UID: "0d3524d6-1b11-41e9-9862-802d687ea87b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:32:44.808256 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.804983 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-sys-fs\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.808256 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-systemd\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.808256 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805029 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-kubelet\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.808256 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805029 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de6e63f3-65e8-4d4d-bad4-015fed1391a3-cni-binary-copy\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.808256 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-etc-kubernetes\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.808256 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805093 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-etc-kubernetes\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.808256 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-cni-bin\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.808256 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805173 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-cni-bin\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.808256 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805198 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-systemd\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.808256 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805214 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-kubelet\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.808256 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysconfig\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.808256 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-cnibin\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.808256 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805266 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-modprobe-d\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.808256 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805290 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysconfig\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.808256 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/560532f2-4a95-4180-a772-b269efd3d73d-etc-tuned\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.808256 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/560532f2-4a95-4180-a772-b269efd3d73d-tmp\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.808895 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfw88\" (UniqueName: \"kubernetes.io/projected/de6e63f3-65e8-4d4d-bad4-015fed1391a3-kube-api-access-qfw88\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.808895 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805410 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-modprobe-d\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.808895 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.805562 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-iptables-alerter-script\") pod \"iptables-alerter-ldwwp\" (UID: \"64dd929e-8d9d-47f3-bcb9-a0371d0fc279\") " pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:32:44.808895 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.808280 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/560532f2-4a95-4180-a772-b269efd3d73d-tmp\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.808895 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.808373 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/560532f2-4a95-4180-a772-b269efd3d73d-etc-tuned\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.808895 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.808428 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovn-node-metrics-cert\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.811624 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.811599 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvtc\" (UniqueName: \"kubernetes.io/projected/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-kube-api-access-8gvtc\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:44.811976 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.811949 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h6qd\" (UniqueName: \"kubernetes.io/projected/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-kube-api-access-2h6qd\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:44.812078 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.811991 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpnf2\" (UniqueName: \"kubernetes.io/projected/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-kube-api-access-xpnf2\") pod \"iptables-alerter-ldwwp\" (UID: \"64dd929e-8d9d-47f3-bcb9-a0371d0fc279\") " pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:32:44.812078 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.812057 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9tcp\" (UniqueName: \"kubernetes.io/projected/560532f2-4a95-4180-a772-b269efd3d73d-kube-api-access-h9tcp\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:44.814659 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.814638 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwqpz\" (UniqueName: \"kubernetes.io/projected/993fd395-dba5-4c79-99d3-72792b8c273c-kube-api-access-zwqpz\") pod \"node-resolver-9fz4x\" (UID: \"993fd395-dba5-4c79-99d3-72792b8c273c\") " pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:32:44.820684 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:44.820661 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:32:44.820684 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:44.820686 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:32:44.820855 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:44.820699 2579 projected.go:196] Error preparing data for projected volume kube-api-access-dn99g for pod openshift-network-diagnostics/network-check-target-6ftrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:32:44.820855 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:44.820788 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g podName:51aeac6f-2315-42d4-b63e-47efd057cf91 nodeName:}" failed. No retries permitted until 2026-10-31 00:32:45.320767688 +0000 UTC m=+2.042082693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dn99g" (UniqueName: "kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g") pod "network-check-target-6ftrv" (UID: "51aeac6f-2315-42d4-b63e-47efd057cf91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:32:44.822047 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.822026 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9pf2\" (UniqueName: \"kubernetes.io/projected/0d3524d6-1b11-41e9-9862-802d687ea87b-kube-api-access-q9pf2\") pod \"network-metrics-daemon-2662j\" (UID: \"0d3524d6-1b11-41e9-9862-802d687ea87b\") " pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:44.822166 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.822148 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfw88\" (UniqueName: \"kubernetes.io/projected/de6e63f3-65e8-4d4d-bad4-015fed1391a3-kube-api-access-qfw88\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:32:44.906686 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.906619 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.906686 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.906667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.906915 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.906705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91f27c45-482c-4a05-998e-18b57ebfa499-host\") pod \"node-ca-n58v5\" (UID: \"91f27c45-482c-4a05-998e-18b57ebfa499\") " pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:32:44.906915 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.906732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n24w\" (UniqueName: \"kubernetes.io/projected/21754645-32d8-4b1b-a7f5-00594f5854f3-kube-api-access-6n24w\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.906915 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.906760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.906915 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.906788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rblj9\" (UniqueName: \"kubernetes.io/projected/91f27c45-482c-4a05-998e-18b57ebfa499-kube-api-access-rblj9\") pod \"node-ca-n58v5\" (UID: \"91f27c45-482c-4a05-998e-18b57ebfa499\") " pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:32:44.906915 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.906798 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91f27c45-482c-4a05-998e-18b57ebfa499-host\") pod \"node-ca-n58v5\" (UID: \"91f27c45-482c-4a05-998e-18b57ebfa499\") " pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:32:44.906915 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.906828 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.906915 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.906840 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.906915 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.906893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-cnibin\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.907257 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.906931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3346ce41-dd23-4e63-8944-43f2d7d4aaec-konnectivity-ca\") pod \"konnectivity-agent-65kvb\" (UID: \"3346ce41-dd23-4e63-8944-43f2d7d4aaec\") " pod="kube-system/konnectivity-agent-65kvb" Oct 31 00:32:44.907257 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.906959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-system-cni-dir\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.907257 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.906982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3346ce41-dd23-4e63-8944-43f2d7d4aaec-agent-certs\") pod \"konnectivity-agent-65kvb\" (UID: \"3346ce41-dd23-4e63-8944-43f2d7d4aaec\") " pod="kube-system/konnectivity-agent-65kvb" Oct 31 00:32:44.907257 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.907031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91f27c45-482c-4a05-998e-18b57ebfa499-serviceca\") pod \"node-ca-n58v5\" (UID: \"91f27c45-482c-4a05-998e-18b57ebfa499\") " pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:32:44.907257 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.907049 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-cnibin\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.907257 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.907103 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-system-cni-dir\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.908087 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.908056 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.908214 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.908089 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.908214 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.908182 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91f27c45-482c-4a05-998e-18b57ebfa499-serviceca\") pod \"node-ca-n58v5\" (UID: \"91f27c45-482c-4a05-998e-18b57ebfa499\") " pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:32:44.908214 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.908193 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:44.908360 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.908220 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3346ce41-dd23-4e63-8944-43f2d7d4aaec-konnectivity-ca\") pod \"konnectivity-agent-65kvb\" (UID: \"3346ce41-dd23-4e63-8944-43f2d7d4aaec\") " pod="kube-system/konnectivity-agent-65kvb" Oct 31 00:32:44.910061 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.910040 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3346ce41-dd23-4e63-8944-43f2d7d4aaec-agent-certs\") pod \"konnectivity-agent-65kvb\" (UID: \"3346ce41-dd23-4e63-8944-43f2d7d4aaec\") " pod="kube-system/konnectivity-agent-65kvb" Oct 31 00:32:44.918668 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.918642 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rblj9\" (UniqueName: \"kubernetes.io/projected/91f27c45-482c-4a05-998e-18b57ebfa499-kube-api-access-rblj9\") pod \"node-ca-n58v5\" (UID: \"91f27c45-482c-4a05-998e-18b57ebfa499\") " pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:32:44.918773 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:44.918706 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n24w\" (UniqueName: \"kubernetes.io/projected/21754645-32d8-4b1b-a7f5-00594f5854f3-kube-api-access-6n24w\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:45.008936 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.008900 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:32:45.029157 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.029126 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6mcgm" Oct 31 00:32:45.039421 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.039398 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:32:45.083476 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.083444 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:32:45.085728 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.085699 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:32:45.088645 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.088628 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:32:45.098201 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:45.098167 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fa48eaf6754635db0f21b009865b16.slice/crio-f2d4327725d3a33a287e14e7d623f858244d368269196a373031c49862b08fde WatchSource:0}: Error finding container f2d4327725d3a33a287e14e7d623f858244d368269196a373031c49862b08fde: Status 404 returned error can't find the container with id f2d4327725d3a33a287e14e7d623f858244d368269196a373031c49862b08fde Oct 31 00:32:45.098738 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:45.098713 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72fb96218f1d813dd9872d369c087ed5.slice/crio-749ca676ba095a8ba9698d1d2464f22a9935fbbbd185907bd44c09179abfde60 WatchSource:0}: Error finding container 749ca676ba095a8ba9698d1d2464f22a9935fbbbd185907bd44c09179abfde60: Status 404 returned error can't find the container with id 749ca676ba095a8ba9698d1d2464f22a9935fbbbd185907bd44c09179abfde60 Oct 31 00:32:45.100079 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:45.100048 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe15ece_2b2c_4a7d_956f_950a1829b2ac.slice/crio-104f041ebdf17becbae4fd9723914460a4894bee6fe87894a6733eaca1791f07 WatchSource:0}: Error finding container 104f041ebdf17becbae4fd9723914460a4894bee6fe87894a6733eaca1791f07: Status 404 returned error can't find the container with id 104f041ebdf17becbae4fd9723914460a4894bee6fe87894a6733eaca1791f07 Oct 31 00:32:45.101428 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:45.101409 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec53026_ab4d_48f6_93cb_b4ce6c95ec03.slice/crio-df08eacd54bfadb4d947b1cc2f064e42c083f6a0048b4612fa5539ca9985b4f6 WatchSource:0}: Error finding container df08eacd54bfadb4d947b1cc2f064e42c083f6a0048b4612fa5539ca9985b4f6: Status 404 returned error can't find the container with id df08eacd54bfadb4d947b1cc2f064e42c083f6a0048b4612fa5539ca9985b4f6 Oct 31 00:32:45.103386 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:45.103364 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993fd395_dba5_4c79_99d3_72792b8c273c.slice/crio-53071f197595f817a55460bb672281104ec03035e16832b108638a3111746883 WatchSource:0}: Error finding container 53071f197595f817a55460bb672281104ec03035e16832b108638a3111746883: Status 404 returned error can't find the container with id 53071f197595f817a55460bb672281104ec03035e16832b108638a3111746883 Oct 31 00:32:45.105360 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.105343 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:32:45.109101 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.109081 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:32:45.118608 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:45.118585 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f27c45_482c_4a05_998e_18b57ebfa499.slice/crio-045eb8964330e018a313575e35948e73f5c1886605fbc9eff322cbe746276a31 WatchSource:0}: Error finding container 045eb8964330e018a313575e35948e73f5c1886605fbc9eff322cbe746276a31: Status 404 returned error can't find the container with id 045eb8964330e018a313575e35948e73f5c1886605fbc9eff322cbe746276a31 Oct 31 00:32:45.155864 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.155831 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-65kvb" Oct 31 00:32:45.162072 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.162048 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:32:45.166762 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:45.166738 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3346ce41_dd23_4e63_8944_43f2d7d4aaec.slice/crio-a311b2965f9d8ee83128a34f312fd8708da25842aef8c59e4e7cd24ea93eed99 WatchSource:0}: Error finding container a311b2965f9d8ee83128a34f312fd8708da25842aef8c59e4e7cd24ea93eed99: Status 404 returned error can't find the container with id a311b2965f9d8ee83128a34f312fd8708da25842aef8c59e4e7cd24ea93eed99 Oct 31 00:32:45.172401 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:45.172374 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21754645_32d8_4b1b_a7f5_00594f5854f3.slice/crio-d5d15af0f24c23067306c0897776d845bd443e7c7ac469369e02d09029476d44 WatchSource:0}: Error finding container d5d15af0f24c23067306c0897776d845bd443e7c7ac469369e02d09029476d44: Status 404 returned error can't find the container with id d5d15af0f24c23067306c0897776d845bd443e7c7ac469369e02d09029476d44 Oct 31 00:32:45.310617 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.310584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:45.310836 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.310629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs\") pod \"network-metrics-daemon-2662j\" (UID: \"0d3524d6-1b11-41e9-9862-802d687ea87b\") " pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:45.310836 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:45.310753 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:32:45.310836 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:45.310763 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:32:45.310836 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:45.310812 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs podName:0d3524d6-1b11-41e9-9862-802d687ea87b nodeName:}" failed. No retries permitted until 2026-10-31 00:32:46.310795671 +0000 UTC m=+3.032110672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs") pod "network-metrics-daemon-2662j" (UID: "0d3524d6-1b11-41e9-9862-802d687ea87b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:32:45.310836 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:45.310829 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret podName:ffd93b08-b857-4571-a1b5-5eaca4456f8b nodeName:}" failed. No retries permitted until 2026-10-31 00:32:46.310820732 +0000 UTC m=+3.032135719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret") pod "global-pull-secret-syncer-7tgpr" (UID: "ffd93b08-b857-4571-a1b5-5eaca4456f8b") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:32:45.411624 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.411534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn99g\" (UniqueName: \"kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g\") pod \"network-check-target-6ftrv\" (UID: \"51aeac6f-2315-42d4-b63e-47efd057cf91\") " pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:32:45.411807 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:45.411702 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:32:45.411807 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:45.411721 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:32:45.411807 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:45.411732 2579 projected.go:196] Error preparing data for projected volume kube-api-access-dn99g for pod openshift-network-diagnostics/network-check-target-6ftrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:32:45.411807 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:45.411792 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g podName:51aeac6f-2315-42d4-b63e-47efd057cf91 nodeName:}" failed. No retries permitted until 2026-10-31 00:32:46.411773259 +0000 UTC m=+3.133088245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dn99g" (UniqueName: "kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g") pod "network-check-target-6ftrv" (UID: "51aeac6f-2315-42d4-b63e-47efd057cf91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:32:45.440788 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.440758 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:32:45.475419 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:45.475390 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64dd929e_8d9d_47f3_bcb9_a0371d0fc279.slice/crio-8a258ac5e07beeb38e563de3d745b46806e9edd50d635777975622c06721a478 WatchSource:0}: Error finding container 8a258ac5e07beeb38e563de3d745b46806e9edd50d635777975622c06721a478: Status 404 returned error can't find the container with id 8a258ac5e07beeb38e563de3d745b46806e9edd50d635777975622c06721a478 Oct 31 00:32:45.554257 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:45.554224 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6e63f3_65e8_4d4d_bad4_015fed1391a3.slice/crio-70578d806cf234fed0c27479459e0a55693f200e9bafce2e118537e5500d9156 WatchSource:0}: Error finding container 70578d806cf234fed0c27479459e0a55693f200e9bafce2e118537e5500d9156: Status 404 returned error can't find the container with id 70578d806cf234fed0c27479459e0a55693f200e9bafce2e118537e5500d9156 Oct 31 00:32:45.564174 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:32:45.564148 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560532f2_4a95_4180_a772_b269efd3d73d.slice/crio-f3f5dda68768af6006f0cb74af7173e7337b390271ab9e50a4f5af65e05ac9f5 WatchSource:0}: Error finding container f3f5dda68768af6006f0cb74af7173e7337b390271ab9e50a4f5af65e05ac9f5: Status 404 returned error can't find the container with id f3f5dda68768af6006f0cb74af7173e7337b390271ab9e50a4f5af65e05ac9f5 Oct 31 00:32:45.572929 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.572906 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:32:45.743991 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.743881 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:27:45 +0000 UTC" deadline="2028-07-20 17:03:55.49983064 +0000 UTC" Oct 31 00:32:45.743991 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.743933 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15088h31m9.755901545s" Oct 31 00:32:45.881597 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.881560 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:45.881806 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:45.881723 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2662j" podUID="0d3524d6-1b11-41e9-9862-802d687ea87b" Oct 31 00:32:45.886149 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.886125 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:32:45.893696 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.893569 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-czkrf" event={"ID":"560532f2-4a95-4180-a772-b269efd3d73d","Type":"ContainerStarted","Data":"f3f5dda68768af6006f0cb74af7173e7337b390271ab9e50a4f5af65e05ac9f5"} Oct 31 00:32:45.895736 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.895652 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hkql2" event={"ID":"21754645-32d8-4b1b-a7f5-00594f5854f3","Type":"ContainerStarted","Data":"d5d15af0f24c23067306c0897776d845bd443e7c7ac469369e02d09029476d44"} Oct 31 00:32:45.898684 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.898617 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n58v5" event={"ID":"91f27c45-482c-4a05-998e-18b57ebfa499","Type":"ContainerStarted","Data":"045eb8964330e018a313575e35948e73f5c1886605fbc9eff322cbe746276a31"} Oct 31 00:32:45.902861 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.902773 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" event={"ID":"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03","Type":"ContainerStarted","Data":"df08eacd54bfadb4d947b1cc2f064e42c083f6a0048b4612fa5539ca9985b4f6"} Oct 31 00:32:45.905967 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.905861 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" event={"ID":"1fe15ece-2b2c-4a7d-956f-950a1829b2ac","Type":"ContainerStarted","Data":"104f041ebdf17becbae4fd9723914460a4894bee6fe87894a6733eaca1791f07"} Oct 31 00:32:45.908662 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.908591 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6mcgm" event={"ID":"de6e63f3-65e8-4d4d-bad4-015fed1391a3","Type":"ContainerStarted","Data":"70578d806cf234fed0c27479459e0a55693f200e9bafce2e118537e5500d9156"} Oct 31 00:32:45.916689 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.916617 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ldwwp" event={"ID":"64dd929e-8d9d-47f3-bcb9-a0371d0fc279","Type":"ContainerStarted","Data":"8a258ac5e07beeb38e563de3d745b46806e9edd50d635777975622c06721a478"} Oct 31 00:32:45.919153 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.919126 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-65kvb" event={"ID":"3346ce41-dd23-4e63-8944-43f2d7d4aaec","Type":"ContainerStarted","Data":"a311b2965f9d8ee83128a34f312fd8708da25842aef8c59e4e7cd24ea93eed99"} Oct 31 00:32:45.925530 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.925499 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9fz4x" event={"ID":"993fd395-dba5-4c79-99d3-72792b8c273c","Type":"ContainerStarted","Data":"53071f197595f817a55460bb672281104ec03035e16832b108638a3111746883"} Oct 31 00:32:45.929766 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.929735 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" event={"ID":"c7fa48eaf6754635db0f21b009865b16","Type":"ContainerStarted","Data":"f2d4327725d3a33a287e14e7d623f858244d368269196a373031c49862b08fde"} Oct 31 00:32:45.942286 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:45.942249 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" event={"ID":"72fb96218f1d813dd9872d369c087ed5","Type":"ContainerStarted","Data":"749ca676ba095a8ba9698d1d2464f22a9935fbbbd185907bd44c09179abfde60"} Oct 31 00:32:46.318747 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:46.318703 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:46.318950 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:46.318799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs\") pod \"network-metrics-daemon-2662j\" (UID: \"0d3524d6-1b11-41e9-9862-802d687ea87b\") " pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:46.323900 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:46.323870 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:32:46.324057 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:46.323972 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret podName:ffd93b08-b857-4571-a1b5-5eaca4456f8b nodeName:}" failed. No retries permitted until 2026-10-31 00:32:48.323950764 +0000 UTC m=+5.045265770 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret") pod "global-pull-secret-syncer-7tgpr" (UID: "ffd93b08-b857-4571-a1b5-5eaca4456f8b") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:32:46.324613 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:46.324594 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:32:46.324699 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:46.324677 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs podName:0d3524d6-1b11-41e9-9862-802d687ea87b nodeName:}" failed. No retries permitted until 2026-10-31 00:32:48.324644735 +0000 UTC m=+5.045959725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs") pod "network-metrics-daemon-2662j" (UID: "0d3524d6-1b11-41e9-9862-802d687ea87b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:32:46.419737 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:46.419693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn99g\" (UniqueName: \"kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g\") pod \"network-check-target-6ftrv\" (UID: \"51aeac6f-2315-42d4-b63e-47efd057cf91\") " pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:32:46.419988 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:46.419970 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:32:46.420065 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:46.419996 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:32:46.420065 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:46.420011 2579 projected.go:196] Error preparing data for projected volume kube-api-access-dn99g for pod openshift-network-diagnostics/network-check-target-6ftrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:32:46.420182 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:46.420071 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g podName:51aeac6f-2315-42d4-b63e-47efd057cf91 nodeName:}" failed. No retries permitted until 2026-10-31 00:32:48.420051843 +0000 UTC m=+5.141366846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dn99g" (UniqueName: "kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g") pod "network-check-target-6ftrv" (UID: "51aeac6f-2315-42d4-b63e-47efd057cf91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:32:46.744719 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:46.744563 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:27:45 +0000 UTC" deadline="2028-08-11 20:36:34.89746533 +0000 UTC" Oct 31 00:32:46.744719 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:46.744634 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15620h3m48.152835635s" Oct 31 00:32:46.881865 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:46.881059 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:46.881865 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:46.881198 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tgpr" podUID="ffd93b08-b857-4571-a1b5-5eaca4456f8b" Oct 31 00:32:46.881865 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:46.881685 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:32:46.881865 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:46.881800 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ftrv" podUID="51aeac6f-2315-42d4-b63e-47efd057cf91" Oct 31 00:32:47.884396 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:47.884366 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:47.884871 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:47.884494 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2662j" podUID="0d3524d6-1b11-41e9-9862-802d687ea87b" Oct 31 00:32:48.343535 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:48.343434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:48.343535 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:48.343506 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs\") pod \"network-metrics-daemon-2662j\" (UID: \"0d3524d6-1b11-41e9-9862-802d687ea87b\") " pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:48.343824 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:48.343724 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:32:48.343824 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:48.343790 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs podName:0d3524d6-1b11-41e9-9862-802d687ea87b nodeName:}" failed. No retries permitted until 2026-10-31 00:32:52.343770517 +0000 UTC m=+9.065085509 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs") pod "network-metrics-daemon-2662j" (UID: "0d3524d6-1b11-41e9-9862-802d687ea87b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:32:48.344244 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:48.344224 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:32:48.344323 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:48.344283 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret podName:ffd93b08-b857-4571-a1b5-5eaca4456f8b nodeName:}" failed. No retries permitted until 2026-10-31 00:32:52.344267945 +0000 UTC m=+9.065582938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret") pod "global-pull-secret-syncer-7tgpr" (UID: "ffd93b08-b857-4571-a1b5-5eaca4456f8b") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:32:48.444785 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:48.444641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn99g\" (UniqueName: \"kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g\") pod \"network-check-target-6ftrv\" (UID: \"51aeac6f-2315-42d4-b63e-47efd057cf91\") " pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:32:48.444983 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:48.444929 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:32:48.444983 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:48.444952 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:32:48.444983 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:48.444965 2579 projected.go:196] Error preparing data for projected volume kube-api-access-dn99g for pod openshift-network-diagnostics/network-check-target-6ftrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:32:48.445152 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:48.445030 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g podName:51aeac6f-2315-42d4-b63e-47efd057cf91 nodeName:}" failed. No retries permitted until 2026-10-31 00:32:52.445009511 +0000 UTC m=+9.166324503 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dn99g" (UniqueName: "kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g") pod "network-check-target-6ftrv" (UID: "51aeac6f-2315-42d4-b63e-47efd057cf91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:32:48.881610 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:48.881531 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:48.881860 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:48.881709 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tgpr" podUID="ffd93b08-b857-4571-a1b5-5eaca4456f8b" Oct 31 00:32:48.882168 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:48.882146 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:32:48.882371 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:48.882250 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ftrv" podUID="51aeac6f-2315-42d4-b63e-47efd057cf91" Oct 31 00:32:49.884668 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:49.883894 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:49.884668 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:49.884223 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2662j" podUID="0d3524d6-1b11-41e9-9862-802d687ea87b" Oct 31 00:32:49.971458 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:49.971422 2579 generic.go:356] "Generic (PLEG): container finished" podID="72fb96218f1d813dd9872d369c087ed5" containerID="5bb9e4deae71ef44feb1d83059ea4d693559fe72c95d79081dba66e319ce1489" exitCode=0 Oct 31 00:32:49.971600 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:49.971523 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" event={"ID":"72fb96218f1d813dd9872d369c087ed5","Type":"ContainerDied","Data":"5bb9e4deae71ef44feb1d83059ea4d693559fe72c95d79081dba66e319ce1489"} Oct 31 00:32:49.978631 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:49.978595 2579 generic.go:356] "Generic (PLEG): container finished" podID="21754645-32d8-4b1b-a7f5-00594f5854f3" containerID="5e74368e51b6d8ab9a0142f25a79bf0153f56b02e388052e910cc0d51480bfac" exitCode=0 Oct 31 00:32:49.978761 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:49.978650 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hkql2" event={"ID":"21754645-32d8-4b1b-a7f5-00594f5854f3","Type":"ContainerDied","Data":"5e74368e51b6d8ab9a0142f25a79bf0153f56b02e388052e910cc0d51480bfac"} Oct 31 00:32:50.881986 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:50.881902 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:50.882173 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:50.882060 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tgpr" podUID="ffd93b08-b857-4571-a1b5-5eaca4456f8b" Oct 31 00:32:50.882173 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:50.882147 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:32:50.882295 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:50.882233 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ftrv" podUID="51aeac6f-2315-42d4-b63e-47efd057cf91" Oct 31 00:32:50.986425 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:50.986356 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-65kvb" event={"ID":"3346ce41-dd23-4e63-8944-43f2d7d4aaec","Type":"ContainerStarted","Data":"02ee88d465cbf5449ca2b89e4c87c9f55503e1c345b71d6addc55c6537559cf6"} Oct 31 00:32:50.990108 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:50.990069 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" event={"ID":"c7fa48eaf6754635db0f21b009865b16","Type":"ContainerStarted","Data":"0af45f65554934b364ab136e274e88d8723f45ec35c849908b023ff88ecb8904"} Oct 31 00:32:50.997153 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:50.997112 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" event={"ID":"72fb96218f1d813dd9872d369c087ed5","Type":"ContainerStarted","Data":"2daad1a885385179bb48c3338384286f0ff687faa81685885b40eae74d9b7ba2"} Oct 31 00:32:51.021773 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:51.021691 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-65kvb" podStartSLOduration=2.223237726 podStartE2EDuration="7.021667403s" totalImagesPullingTime="4.798429677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:51.006458951 +0000 UTC m=+7.727773964" watchObservedRunningTime="2026-10-31 00:32:51.021667403 +0000 UTC m=+7.742982416" Oct 31 00:32:51.039642 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:51.039573 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" podStartSLOduration=7.039550904 podStartE2EDuration="7.039550904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:51.022532852 +0000 UTC m=+7.743847864" watchObservedRunningTime="2026-10-31 00:32:51.039550904 +0000 UTC m=+7.760865917" Oct 31 00:32:51.882890 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:51.882853 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:51.883059 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:51.883019 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2662j" podUID="0d3524d6-1b11-41e9-9862-802d687ea87b" Oct 31 00:32:52.006320 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:52.006161 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n58v5" event={"ID":"91f27c45-482c-4a05-998e-18b57ebfa499","Type":"ContainerStarted","Data":"013c505d19a25aac87bc9b9ed8b3064fcd7fe1e9b28d2aae82fcc2c28fdf0cfd"} Oct 31 00:32:52.023998 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:52.023950 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" podStartSLOduration=8.023934862 podStartE2EDuration="8.023934862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:51.03990689 +0000 UTC m=+7.761221903" watchObservedRunningTime="2026-10-31 00:32:52.023934862 +0000 UTC m=+8.745249872" Oct 31 00:32:52.024287 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:52.024255 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n58v5" podStartSLOduration=1.503977616 podStartE2EDuration="8.024248557s" totalImagesPullingTime="6.520270941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:52.023442628 +0000 UTC m=+8.744757660" watchObservedRunningTime="2026-10-31 00:32:52.024248557 +0000 UTC m=+8.745563569" Oct 31 00:32:52.382788 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:52.382747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:52.383004 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:52.382813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs\") pod \"network-metrics-daemon-2662j\" (UID: \"0d3524d6-1b11-41e9-9862-802d687ea87b\") " pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:52.383004 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:52.382974 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:32:52.383126 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:52.383040 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs podName:0d3524d6-1b11-41e9-9862-802d687ea87b nodeName:}" failed. No retries permitted until 2026-10-31 00:33:00.383021526 +0000 UTC m=+17.104336518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs") pod "network-metrics-daemon-2662j" (UID: "0d3524d6-1b11-41e9-9862-802d687ea87b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:32:52.383473 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:52.383455 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:32:52.383564 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:52.383511 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret podName:ffd93b08-b857-4571-a1b5-5eaca4456f8b nodeName:}" failed. No retries permitted until 2026-10-31 00:33:00.383494302 +0000 UTC m=+17.104809295 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret") pod "global-pull-secret-syncer-7tgpr" (UID: "ffd93b08-b857-4571-a1b5-5eaca4456f8b") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:32:52.483906 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:52.483805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn99g\" (UniqueName: \"kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g\") pod \"network-check-target-6ftrv\" (UID: \"51aeac6f-2315-42d4-b63e-47efd057cf91\") " pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:32:52.484083 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:52.483997 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:32:52.484083 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:52.484065 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:32:52.484182 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:52.484131 2579 projected.go:196] Error preparing data for projected volume kube-api-access-dn99g for pod openshift-network-diagnostics/network-check-target-6ftrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:32:52.484217 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:52.484209 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g podName:51aeac6f-2315-42d4-b63e-47efd057cf91 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:00.484187443 +0000 UTC m=+17.205502445 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dn99g" (UniqueName: "kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g") pod "network-check-target-6ftrv" (UID: "51aeac6f-2315-42d4-b63e-47efd057cf91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:32:52.765078 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:52.764983 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-65kvb" Oct 31 00:32:52.765867 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:52.765844 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-65kvb" Oct 31 00:32:52.881695 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:52.881639 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:52.881883 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:52.881812 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tgpr" podUID="ffd93b08-b857-4571-a1b5-5eaca4456f8b" Oct 31 00:32:52.882333 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:52.882311 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:32:52.882435 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:52.882407 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ftrv" podUID="51aeac6f-2315-42d4-b63e-47efd057cf91" Oct 31 00:32:53.010547 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:53.010509 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" event={"ID":"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03","Type":"ContainerStarted","Data":"74c1dffabebed15c1c1867a76d205dfec1fa213d2b2c9085c40cda72c6fdaf8d"} Oct 31 00:32:53.883199 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:53.883164 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:53.883382 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:53.883307 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2662j" podUID="0d3524d6-1b11-41e9-9862-802d687ea87b" Oct 31 00:32:53.991967 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:53.991941 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:32:54.015984 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:54.015956 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:32:54.016410 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:54.016342 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" event={"ID":"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03","Type":"ContainerStarted","Data":"d697919acaed000fd0f7840d8e37b739d1cd3b8d01ab5287abb9bded2544e783"} Oct 31 00:32:54.881882 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:54.881847 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:32:54.882056 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:54.881983 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ftrv" podUID="51aeac6f-2315-42d4-b63e-47efd057cf91" Oct 31 00:32:54.882333 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:54.882313 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:54.882440 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:54.882419 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tgpr" podUID="ffd93b08-b857-4571-a1b5-5eaca4456f8b" Oct 31 00:32:54.896351 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:54.895304 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:32:53.991963007Z","UUID":"5fbd8e37-d898-4e37-82e6-2139cb48dfda","Handler":null,"Name":"","Endpoint":""} Oct 31 00:32:54.898272 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:54.897743 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:32:54.898272 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:54.897770 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:32:55.021103 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:55.021020 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6mcgm" event={"ID":"de6e63f3-65e8-4d4d-bad4-015fed1391a3","Type":"ContainerStarted","Data":"1180fc79d1f540a4aec5aed7df30759404bedf24e4f8c8d4decbd742e21ef902"} Oct 31 00:32:55.025924 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:55.025886 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9fz4x" event={"ID":"993fd395-dba5-4c79-99d3-72792b8c273c","Type":"ContainerStarted","Data":"1869bfd9e3705c824962ac271b1973821c768d3462be98e5c1847fd41fee2049"} Oct 31 00:32:55.028473 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:55.028444 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hkql2" event={"ID":"21754645-32d8-4b1b-a7f5-00594f5854f3","Type":"ContainerStarted","Data":"fcb6de33bc4e33ea05105d6197f8213edd3e409088440d0694311cdce30afe62"} Oct 31 00:32:55.039203 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:55.039135 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-6mcgm" podStartSLOduration=2.078669762 podStartE2EDuration="11.039117423s" totalImagesPullingTime="8.960447661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:55.038848247 +0000 UTC m=+11.760163256" watchObservedRunningTime="2026-10-31 00:32:55.039117423 +0000 UTC m=+11.760432452" Oct 31 00:32:55.073336 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:55.073273 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9fz4x" podStartSLOduration=1.778876905 podStartE2EDuration="11.073254999s" totalImagesPullingTime="9.294378094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:55.072788648 +0000 UTC m=+11.794103656" watchObservedRunningTime="2026-10-31 00:32:55.073254999 +0000 UTC m=+11.794570011" Oct 31 00:32:55.886292 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:55.886009 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:55.886494 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:55.886367 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2662j" podUID="0d3524d6-1b11-41e9-9862-802d687ea87b" Oct 31 00:32:56.033121 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:56.033084 2579 generic.go:356] "Generic (PLEG): container finished" podID="21754645-32d8-4b1b-a7f5-00594f5854f3" containerID="fcb6de33bc4e33ea05105d6197f8213edd3e409088440d0694311cdce30afe62" exitCode=0 Oct 31 00:32:56.033594 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:56.033171 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hkql2" event={"ID":"21754645-32d8-4b1b-a7f5-00594f5854f3","Type":"ContainerDied","Data":"fcb6de33bc4e33ea05105d6197f8213edd3e409088440d0694311cdce30afe62"} Oct 31 00:32:56.036532 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:56.036480 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" event={"ID":"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03","Type":"ContainerStarted","Data":"3765e73e51af6c95c1c8b63b7d1c834ed9fbea9439be372a488eff0e1deb4c17"} Oct 31 00:32:56.071437 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:56.071335 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" podStartSLOduration=2.450002041 podStartE2EDuration="12.071317318s" totalImagesPullingTime="9.621315277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:32:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:56.070983996 +0000 UTC m=+12.792299004" watchObservedRunningTime="2026-10-31 00:32:56.071317318 +0000 UTC m=+12.792632354" Oct 31 00:32:56.881623 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:56.881589 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:56.881744 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:56.881663 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:32:56.881806 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:56.881765 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tgpr" podUID="ffd93b08-b857-4571-a1b5-5eaca4456f8b" Oct 31 00:32:56.881875 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:56.881855 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ftrv" podUID="51aeac6f-2315-42d4-b63e-47efd057cf91" Oct 31 00:32:57.040655 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:57.040600 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-czkrf" event={"ID":"560532f2-4a95-4180-a772-b269efd3d73d","Type":"ContainerStarted","Data":"80b4e1e3d0d6828333664c6323aff51746d948f41b4653a81a581ab98a6fb6a6"} Oct 31 00:32:57.042402 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:57.042374 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ldwwp" event={"ID":"64dd929e-8d9d-47f3-bcb9-a0371d0fc279","Type":"ContainerStarted","Data":"eaa5a2861f0447d096f4bcba2d538ae5f03b8bbc405b703fecbf8b1524e80925"} Oct 31 00:32:57.058678 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:57.058555 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-czkrf" podStartSLOduration=1.823091522 podStartE2EDuration="13.058539159s" totalImagesPullingTime="11.235447637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:57.058101055 +0000 UTC m=+13.779416063" watchObservedRunningTime="2026-10-31 00:32:57.058539159 +0000 UTC m=+13.779854208" Oct 31 00:32:57.071674 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:57.071617 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ldwwp" podStartSLOduration=3.7926072 podStartE2EDuration="13.071599551s" totalImagesPullingTime="9.278992351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:57.070948254 +0000 UTC m=+13.792263262" watchObservedRunningTime="2026-10-31 00:32:57.071599551 +0000 UTC m=+13.792914560" Oct 31 00:32:57.884873 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:57.884839 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:57.885046 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:57.884943 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2662j" podUID="0d3524d6-1b11-41e9-9862-802d687ea87b" Oct 31 00:32:58.046980 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:58.046948 2579 generic.go:356] "Generic (PLEG): container finished" podID="21754645-32d8-4b1b-a7f5-00594f5854f3" containerID="595d3eaf77f1a61c9461bceaeafa2899e0387d9a72766ccddbe60260f98f253d" exitCode=0 Oct 31 00:32:58.047530 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:58.047045 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hkql2" event={"ID":"21754645-32d8-4b1b-a7f5-00594f5854f3","Type":"ContainerDied","Data":"595d3eaf77f1a61c9461bceaeafa2899e0387d9a72766ccddbe60260f98f253d"} Oct 31 00:32:58.882020 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:58.881981 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:32:58.882211 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:58.882031 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:32:58.882211 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:58.882122 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ftrv" podUID="51aeac6f-2315-42d4-b63e-47efd057cf91" Oct 31 00:32:58.882285 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:58.882262 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tgpr" podUID="ffd93b08-b857-4571-a1b5-5eaca4456f8b" Oct 31 00:32:59.886524 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:32:59.886311 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:32:59.887130 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:32:59.886656 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2662j" podUID="0d3524d6-1b11-41e9-9862-802d687ea87b" Oct 31 00:33:00.022047 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:00.022011 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-65kvb" Oct 31 00:33:00.022233 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:00.022148 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:33:00.022803 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:00.022784 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-65kvb" Oct 31 00:33:00.053547 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:00.053498 2579 generic.go:356] "Generic (PLEG): container finished" podID="21754645-32d8-4b1b-a7f5-00594f5854f3" containerID="d9610eb58a1faf07f1cf7d046b7cbdea77cb76b00f98a75813c041122fcac570" exitCode=0 Oct 31 00:33:00.053719 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:00.053563 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hkql2" event={"ID":"21754645-32d8-4b1b-a7f5-00594f5854f3","Type":"ContainerDied","Data":"d9610eb58a1faf07f1cf7d046b7cbdea77cb76b00f98a75813c041122fcac570"} Oct 31 00:33:00.450445 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:00.450402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:00.450619 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:00.450457 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs\") pod \"network-metrics-daemon-2662j\" (UID: \"0d3524d6-1b11-41e9-9862-802d687ea87b\") " pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:00.450619 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:00.450564 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:33:00.450619 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:00.450571 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:33:00.450782 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:00.450630 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret podName:ffd93b08-b857-4571-a1b5-5eaca4456f8b nodeName:}" failed. No retries permitted until 2026-10-31 00:33:16.450616111 +0000 UTC m=+33.171931108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret") pod "global-pull-secret-syncer-7tgpr" (UID: "ffd93b08-b857-4571-a1b5-5eaca4456f8b") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:33:00.450782 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:00.450660 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs podName:0d3524d6-1b11-41e9-9862-802d687ea87b nodeName:}" failed. No retries permitted until 2026-10-31 00:33:16.450651398 +0000 UTC m=+33.171966396 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs") pod "network-metrics-daemon-2662j" (UID: "0d3524d6-1b11-41e9-9862-802d687ea87b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:33:00.551304 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:00.551264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn99g\" (UniqueName: \"kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g\") pod \"network-check-target-6ftrv\" (UID: \"51aeac6f-2315-42d4-b63e-47efd057cf91\") " pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:00.551453 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:00.551431 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:33:00.551453 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:00.551453 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:33:00.551545 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:00.551465 2579 projected.go:196] Error preparing data for projected volume kube-api-access-dn99g for pod openshift-network-diagnostics/network-check-target-6ftrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:33:00.551545 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:00.551522 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g podName:51aeac6f-2315-42d4-b63e-47efd057cf91 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:16.551509372 +0000 UTC m=+33.272824375 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dn99g" (UniqueName: "kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g") pod "network-check-target-6ftrv" (UID: "51aeac6f-2315-42d4-b63e-47efd057cf91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:33:00.881759 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:00.881682 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:00.881908 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:00.881819 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tgpr" podUID="ffd93b08-b857-4571-a1b5-5eaca4456f8b" Oct 31 00:33:00.881908 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:00.881869 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:00.882026 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:00.881960 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ftrv" podUID="51aeac6f-2315-42d4-b63e-47efd057cf91" Oct 31 00:33:01.881586 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:01.881327 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:01.882080 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:01.881662 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2662j" podUID="0d3524d6-1b11-41e9-9862-802d687ea87b" Oct 31 00:33:02.060535 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:02.060504 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" event={"ID":"1fe15ece-2b2c-4a7d-956f-950a1829b2ac","Type":"ContainerStarted","Data":"981bd3a3cecc7724edfdb1054ba1f89674999e2d00e67de4e1bb840c063fe472"} Oct 31 00:33:02.060666 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:02.060599 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" event={"ID":"1fe15ece-2b2c-4a7d-956f-950a1829b2ac","Type":"ContainerStarted","Data":"09ed6a7b463d7286bb392d44aecc6865b0c79a39a70302744318abcd45a93737"} Oct 31 00:33:02.061473 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:02.061010 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" event={"ID":"1fe15ece-2b2c-4a7d-956f-950a1829b2ac","Type":"ContainerStarted","Data":"6ece0033dd6dbbf5282d7939e23aa5139313710e348e00a067bb1abf453a201a"} Oct 31 00:33:02.061578 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:02.061559 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" event={"ID":"1fe15ece-2b2c-4a7d-956f-950a1829b2ac","Type":"ContainerStarted","Data":"8cd7c2a8119ec817b9d9353c08fe01c928c5a989548eb53475ddd56d93f94ec1"} Oct 31 00:33:02.061672 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:02.061655 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" event={"ID":"1fe15ece-2b2c-4a7d-956f-950a1829b2ac","Type":"ContainerStarted","Data":"a344fa9544a8bdc5a8c178630d649d4c1529fce4f81fe13b975d7355ec3fb830"} Oct 31 00:33:02.061743 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:02.061731 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" event={"ID":"1fe15ece-2b2c-4a7d-956f-950a1829b2ac","Type":"ContainerStarted","Data":"a78e44d4ad36bfe8d2093de68f0888ddccc95610ecae1af98e5d876eb3190c2e"} Oct 31 00:33:02.881455 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:02.881418 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:02.881455 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:02.881452 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:02.882145 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:02.881511 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ftrv" podUID="51aeac6f-2315-42d4-b63e-47efd057cf91" Oct 31 00:33:02.882145 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:02.881566 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tgpr" podUID="ffd93b08-b857-4571-a1b5-5eaca4456f8b" Oct 31 00:33:03.064953 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:03.064918 2579 generic.go:356] "Generic (PLEG): container finished" podID="21754645-32d8-4b1b-a7f5-00594f5854f3" containerID="81feb16766bf180f56656b1256f6e2fac4afe6605d7907972e31a554f901d185" exitCode=0 Oct 31 00:33:03.065111 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:03.064969 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hkql2" event={"ID":"21754645-32d8-4b1b-a7f5-00594f5854f3","Type":"ContainerDied","Data":"81feb16766bf180f56656b1256f6e2fac4afe6605d7907972e31a554f901d185"} Oct 31 00:33:03.882083 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:03.882056 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:03.882826 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:03.882151 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2662j" podUID="0d3524d6-1b11-41e9-9862-802d687ea87b" Oct 31 00:33:04.070243 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:04.070158 2579 generic.go:356] "Generic (PLEG): container finished" podID="21754645-32d8-4b1b-a7f5-00594f5854f3" containerID="22dcc44dadb68207d3ed96663e3a7988bcc86794af6eb527e491915e9965f1b9" exitCode=0 Oct 31 00:33:04.070387 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:04.070241 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hkql2" event={"ID":"21754645-32d8-4b1b-a7f5-00594f5854f3","Type":"ContainerDied","Data":"22dcc44dadb68207d3ed96663e3a7988bcc86794af6eb527e491915e9965f1b9"} Oct 31 00:33:04.073450 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:04.073423 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" event={"ID":"1fe15ece-2b2c-4a7d-956f-950a1829b2ac","Type":"ContainerStarted","Data":"45a3828495aeccb16b5c2b1cf55d9e58810451fb0dc2d92363f75ffb531a5459"} Oct 31 00:33:04.881182 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:04.881134 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:04.881340 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:04.881164 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:04.881340 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:04.881252 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ftrv" podUID="51aeac6f-2315-42d4-b63e-47efd057cf91" Oct 31 00:33:04.881340 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:04.881325 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tgpr" podUID="ffd93b08-b857-4571-a1b5-5eaca4456f8b" Oct 31 00:33:05.078368 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:05.078337 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hkql2" event={"ID":"21754645-32d8-4b1b-a7f5-00594f5854f3","Type":"ContainerStarted","Data":"b05d8134ed8be19d4edc097766bc147c527e04eef51ecee4b5c049ea99509379"} Oct 31 00:33:05.101359 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:05.101315 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hkql2" podStartSLOduration=8.572886066 podStartE2EDuration="21.101302322s" totalImagesPullingTime="12.528416256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:32:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:05.100754458 +0000 UTC m=+21.822069466" watchObservedRunningTime="2026-10-31 00:33:05.101302322 +0000 UTC m=+21.822617330" Oct 31 00:33:05.881607 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:05.881577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:05.881772 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:05.881748 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2662j" podUID="0d3524d6-1b11-41e9-9862-802d687ea87b" Oct 31 00:33:06.084722 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:06.084509 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" event={"ID":"1fe15ece-2b2c-4a7d-956f-950a1829b2ac","Type":"ContainerStarted","Data":"035643c7481bdcc91e88aac04287f28e18be94fbf130a476f37b3621be62117e"} Oct 31 00:33:06.113871 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:06.113744 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" podStartSLOduration=5.874466783 podStartE2EDuration="22.113730012s" totalImagesPullingTime="16.239263229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:06.112436914 +0000 UTC m=+22.833751933" watchObservedRunningTime="2026-10-31 00:33:06.113730012 +0000 UTC m=+22.835045018" Oct 31 00:33:06.881457 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:06.881422 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:06.881608 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:06.881421 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:06.881608 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:06.881531 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tgpr" podUID="ffd93b08-b857-4571-a1b5-5eaca4456f8b" Oct 31 00:33:06.881720 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:06.881620 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ftrv" podUID="51aeac6f-2315-42d4-b63e-47efd057cf91" Oct 31 00:33:07.087648 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:07.087575 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:07.088154 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:07.087701 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:07.088154 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:07.087751 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:07.109908 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:07.109878 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:07.111154 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:07.111136 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:07.614066 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:07.613982 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7tgpr"] Oct 31 00:33:07.614221 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:07.614109 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:07.614263 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:07.614230 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tgpr" podUID="ffd93b08-b857-4571-a1b5-5eaca4456f8b" Oct 31 00:33:07.614539 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:07.614516 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6ftrv"] Oct 31 00:33:07.614627 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:07.614615 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:07.614752 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:07.614732 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ftrv" podUID="51aeac6f-2315-42d4-b63e-47efd057cf91" Oct 31 00:33:07.615095 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:07.615079 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2662j"] Oct 31 00:33:07.615167 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:07.615156 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:07.615259 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:07.615243 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2662j" podUID="0d3524d6-1b11-41e9-9862-802d687ea87b" Oct 31 00:33:08.881546 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:08.881507 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:08.882082 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:08.881618 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:08.882082 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:08.881657 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:08.882082 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:08.881652 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ftrv" podUID="51aeac6f-2315-42d4-b63e-47efd057cf91" Oct 31 00:33:08.882082 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:08.881744 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tgpr" podUID="ffd93b08-b857-4571-a1b5-5eaca4456f8b" Oct 31 00:33:08.882082 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:08.881830 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2662j" podUID="0d3524d6-1b11-41e9-9862-802d687ea87b" Oct 31 00:33:08.888600 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:08.888565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:09.037648 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:09.037612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:09.126185 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:09.126148 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:09.226881 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:09.226791 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:10.881497 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:10.881406 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:10.881497 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:10.881452 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:10.881497 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:10.881453 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:10.882057 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:10.881535 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tgpr" podUID="ffd93b08-b857-4571-a1b5-5eaca4456f8b" Oct 31 00:33:10.882057 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:10.881612 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ftrv" podUID="51aeac6f-2315-42d4-b63e-47efd057cf91" Oct 31 00:33:10.882057 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:10.881716 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2662j" podUID="0d3524d6-1b11-41e9-9862-802d687ea87b" Oct 31 00:33:12.803466 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:12.803429 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:12.881681 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:12.881483 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:12.881841 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:12.881483 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:12.881841 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:12.881759 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7tgpr" podUID="ffd93b08-b857-4571-a1b5-5eaca4456f8b" Oct 31 00:33:12.881924 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:12.881855 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ftrv" podUID="51aeac6f-2315-42d4-b63e-47efd057cf91" Oct 31 00:33:12.881924 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:12.881483 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:12.881993 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:12.881973 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2662j" podUID="0d3524d6-1b11-41e9-9862-802d687ea87b" Oct 31 00:33:14.036413 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.036381 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeReady" Oct 31 00:33:14.036954 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.036489 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 00:33:14.084397 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.084362 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vdxdv"] Oct 31 00:33:14.087842 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.087820 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vdxdv" Oct 31 00:33:14.091451 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.091428 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-72zjc"] Oct 31 00:33:14.095286 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.095266 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:14.097969 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.097946 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vdxdv"] Oct 31 00:33:14.098440 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.098358 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:33:14.098562 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.098522 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:33:14.098618 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.098585 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hwf9w\"" Oct 31 00:33:14.098723 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.098702 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:33:14.098836 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.098734 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:33:14.101431 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.101414 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kh74f\"" Oct 31 00:33:14.103455 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.103432 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-72zjc"] Oct 31 00:33:14.180987 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.180914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:14.197343 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.197318 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:33:14.236578 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.236550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:14.236721 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.236587 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssv6w\" (UniqueName: \"kubernetes.io/projected/9f037fa6-1d56-42b5-922e-007f1af8552c-kube-api-access-ssv6w\") pod \"ingress-canary-vdxdv\" (UID: \"9f037fa6-1d56-42b5-922e-007f1af8552c\") " pod="openshift-ingress-canary/ingress-canary-vdxdv" Oct 31 00:33:14.236759 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.236746 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/330ec43b-421e-42ee-b61e-e26743f9fbb7-tmp-dir\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:14.236792 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.236780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/330ec43b-421e-42ee-b61e-e26743f9fbb7-config-volume\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:14.236820 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.236798 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm9r6\" (UniqueName: \"kubernetes.io/projected/330ec43b-421e-42ee-b61e-e26743f9fbb7-kube-api-access-sm9r6\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:14.236856 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.236820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f037fa6-1d56-42b5-922e-007f1af8552c-cert\") pod \"ingress-canary-vdxdv\" (UID: \"9f037fa6-1d56-42b5-922e-007f1af8552c\") " pod="openshift-ingress-canary/ingress-canary-vdxdv" Oct 31 00:33:14.337587 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.337547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/330ec43b-421e-42ee-b61e-e26743f9fbb7-tmp-dir\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:14.337587 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.337590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/330ec43b-421e-42ee-b61e-e26743f9fbb7-config-volume\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:14.337798 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.337605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm9r6\" (UniqueName: \"kubernetes.io/projected/330ec43b-421e-42ee-b61e-e26743f9fbb7-kube-api-access-sm9r6\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:14.337798 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.337689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f037fa6-1d56-42b5-922e-007f1af8552c-cert\") pod \"ingress-canary-vdxdv\" (UID: \"9f037fa6-1d56-42b5-922e-007f1af8552c\") " pod="openshift-ingress-canary/ingress-canary-vdxdv" Oct 31 00:33:14.337798 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.337748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:14.337798 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.337777 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssv6w\" (UniqueName: \"kubernetes.io/projected/9f037fa6-1d56-42b5-922e-007f1af8552c-kube-api-access-ssv6w\") pod \"ingress-canary-vdxdv\" (UID: \"9f037fa6-1d56-42b5-922e-007f1af8552c\") " pod="openshift-ingress-canary/ingress-canary-vdxdv" Oct 31 00:33:14.337984 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:14.337866 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:33:14.337984 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:14.337929 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls podName:330ec43b-421e-42ee-b61e-e26743f9fbb7 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:14.837909004 +0000 UTC m=+31.559224003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls") pod "dns-default-72zjc" (UID: "330ec43b-421e-42ee-b61e-e26743f9fbb7") : secret "dns-default-metrics-tls" not found Oct 31 00:33:14.338063 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.337983 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/330ec43b-421e-42ee-b61e-e26743f9fbb7-tmp-dir\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:14.338352 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.338330 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/330ec43b-421e-42ee-b61e-e26743f9fbb7-config-volume\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:14.342016 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.341995 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f037fa6-1d56-42b5-922e-007f1af8552c-cert\") pod \"ingress-canary-vdxdv\" (UID: \"9f037fa6-1d56-42b5-922e-007f1af8552c\") " pod="openshift-ingress-canary/ingress-canary-vdxdv" Oct 31 00:33:14.346619 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.346597 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm9r6\" (UniqueName: \"kubernetes.io/projected/330ec43b-421e-42ee-b61e-e26743f9fbb7-kube-api-access-sm9r6\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:14.347079 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.347059 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssv6w\" (UniqueName: \"kubernetes.io/projected/9f037fa6-1d56-42b5-922e-007f1af8552c-kube-api-access-ssv6w\") pod \"ingress-canary-vdxdv\" (UID: \"9f037fa6-1d56-42b5-922e-007f1af8552c\") " pod="openshift-ingress-canary/ingress-canary-vdxdv" Oct 31 00:33:14.400409 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.400377 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vdxdv" Oct 31 00:33:14.683076 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.683039 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vdxdv"] Oct 31 00:33:14.686737 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:33:14.686704 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f037fa6_1d56_42b5_922e_007f1af8552c.slice/crio-2367e34dfbf292d0fff1e7b7cec1f944113d6acd7f49a860a725b7404cc10514 WatchSource:0}: Error finding container 2367e34dfbf292d0fff1e7b7cec1f944113d6acd7f49a860a725b7404cc10514: Status 404 returned error can't find the container with id 2367e34dfbf292d0fff1e7b7cec1f944113d6acd7f49a860a725b7404cc10514 Oct 31 00:33:14.841590 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.841509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:14.841756 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:14.841704 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:33:14.841794 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:14.841776 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls podName:330ec43b-421e-42ee-b61e-e26743f9fbb7 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:15.841761022 +0000 UTC m=+32.563076006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls") pod "dns-default-72zjc" (UID: "330ec43b-421e-42ee-b61e-e26743f9fbb7") : secret "dns-default-metrics-tls" not found Oct 31 00:33:14.881407 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.881369 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:14.881407 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.881400 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:14.881407 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.881414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:14.885122 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.885094 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:33:14.885122 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.885092 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:33:14.886084 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.886065 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:33:14.886198 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.886179 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-87knn\"" Oct 31 00:33:14.886253 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.886206 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v94lw\"" Oct 31 00:33:14.886253 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.886237 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:33:14.886328 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:14.886314 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qhb2k\"" Oct 31 00:33:15.105369 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:15.105285 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vdxdv" event={"ID":"9f037fa6-1d56-42b5-922e-007f1af8552c","Type":"ContainerStarted","Data":"2367e34dfbf292d0fff1e7b7cec1f944113d6acd7f49a860a725b7404cc10514"} Oct 31 00:33:15.665194 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:15.665155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:15.750303 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:15.750268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:15.849577 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:15.849536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:15.849777 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:15.849712 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:33:15.849848 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:15.849791 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls podName:330ec43b-421e-42ee-b61e-e26743f9fbb7 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:17.849771749 +0000 UTC m=+34.571086750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls") pod "dns-default-72zjc" (UID: "330ec43b-421e-42ee-b61e-e26743f9fbb7") : secret "dns-default-metrics-tls" not found Oct 31 00:33:15.888782 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:15.888751 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:15.930598 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:15.930511 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pgt89"] Oct 31 00:33:15.954400 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:15.954056 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:15.954585 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:15.954068 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pgt89"] Oct 31 00:33:15.960870 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:15.960816 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-65vdk-w4x2d"] Oct 31 00:33:15.963012 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:15.962985 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:33:15.963012 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:15.962985 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:33:15.963182 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:15.963086 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:33:15.966250 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:15.965875 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6p62k\"" Oct 31 00:33:15.977874 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:15.977846 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-65vdk-w4x2d"] Oct 31 00:33:15.978041 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:15.977969 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:15.983418 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:15.983197 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-l8vbc"] Oct 31 00:33:15.999946 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:15.999917 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-l8vbc"] Oct 31 00:33:16.000081 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.000035 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l8vbc" Oct 31 00:33:16.002591 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.001486 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:16.051444 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.051415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:16.051595 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.051477 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:16.051595 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.051528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-archives-path\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:16.051725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.051596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bbc352c4-a2e0-496b-8a9e-56b10e016a74-data-volume\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:16.051725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.051623 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-trusted-ca-bundle\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:16.051725 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.051662 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45lrv\" (UniqueName: \"kubernetes.io/projected/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-api-access-45lrv\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:16.051815 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.051757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qcz5\" (UniqueName: \"kubernetes.io/projected/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-kube-api-access-8qcz5\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:16.051815 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.051772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bbc352c4-a2e0-496b-8a9e-56b10e016a74-crio-socket\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:16.051815 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.051791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-service-ca-bundle\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:16.058573 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.058550 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-x5t4j\"" Oct 31 00:33:16.058709 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.058619 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pqlnb\"" Oct 31 00:33:16.059327 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.059303 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 00:33:16.064273 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.064250 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:33:16.064407 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.064311 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 00:33:16.064490 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.064442 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 00:33:16.064490 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.064468 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 00:33:16.095092 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.095055 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:16.131266 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.131233 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-f9mbc"] Oct 31 00:33:16.150516 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.150483 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-f9mbc"] Oct 31 00:33:16.150678 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.150586 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" Oct 31 00:33:16.152212 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.152187 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bbc352c4-a2e0-496b-8a9e-56b10e016a74-data-volume\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:16.152337 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.152216 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-trusted-ca-bundle\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:16.152337 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.152243 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfg9f\" (UniqueName: \"kubernetes.io/projected/1a6cf89c-e10a-4a46-a62f-489113be6ac4-kube-api-access-gfg9f\") pod \"migrator-66df5d496b-l8vbc\" (UID: \"1a6cf89c-e10a-4a46-a62f-489113be6ac4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l8vbc" Oct 31 00:33:16.152337 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.152275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45lrv\" (UniqueName: \"kubernetes.io/projected/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-api-access-45lrv\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:16.152461 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.152354 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qcz5\" (UniqueName: \"kubernetes.io/projected/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-kube-api-access-8qcz5\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:16.152461 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.152392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bbc352c4-a2e0-496b-8a9e-56b10e016a74-crio-socket\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:16.152461 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.152426 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-service-ca-bundle\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:16.152461 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.152456 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:16.152623 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.152514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:16.152623 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.152545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-archives-path\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:16.152623 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.152561 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bbc352c4-a2e0-496b-8a9e-56b10e016a74-data-volume\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:16.152783 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:16.152622 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:33:16.152783 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.152660 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bbc352c4-a2e0-496b-8a9e-56b10e016a74-crio-socket\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:16.152783 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:16.152703 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm podName:bbc352c4-a2e0-496b-8a9e-56b10e016a74 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:16.652684228 +0000 UTC m=+33.373999215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgt89" (UID: "bbc352c4-a2e0-496b-8a9e-56b10e016a74") : configmap "kube-rbac-proxy" not found Oct 31 00:33:16.152783 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:16.152728 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:33:16.152783 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:16.152781 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls podName:bbc352c4-a2e0-496b-8a9e-56b10e016a74 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:16.652766418 +0000 UTC m=+33.374081410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgt89" (UID: "bbc352c4-a2e0-496b-8a9e-56b10e016a74") : secret "insights-runtime-extractor-tls" not found Oct 31 00:33:16.152963 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.152867 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-archives-path\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:16.153024 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.153007 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-service-ca-bundle\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:16.153279 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.153264 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-trusted-ca-bundle\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:16.161171 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.161143 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45lrv\" (UniqueName: \"kubernetes.io/projected/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-api-access-45lrv\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:16.161291 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.161158 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qcz5\" (UniqueName: \"kubernetes.io/projected/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-kube-api-access-8qcz5\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:16.162193 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.162169 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 00:33:16.162193 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.162189 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jbsnc\"" Oct 31 00:33:16.162411 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.162229 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 00:33:16.162411 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.162243 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 00:33:16.162411 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.162269 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 00:33:16.195865 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.195780 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:16.253784 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.253749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw8cx\" (UniqueName: \"kubernetes.io/projected/c48c473c-b40e-452f-9e50-5008cddef4b4-kube-api-access-rw8cx\") pod \"service-ca-5fffc97bdf-f9mbc\" (UID: \"c48c473c-b40e-452f-9e50-5008cddef4b4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" Oct 31 00:33:16.253955 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.253819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c48c473c-b40e-452f-9e50-5008cddef4b4-signing-cabundle\") pod \"service-ca-5fffc97bdf-f9mbc\" (UID: \"c48c473c-b40e-452f-9e50-5008cddef4b4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" Oct 31 00:33:16.254036 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.254011 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c48c473c-b40e-452f-9e50-5008cddef4b4-signing-key\") pod \"service-ca-5fffc97bdf-f9mbc\" (UID: \"c48c473c-b40e-452f-9e50-5008cddef4b4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" Oct 31 00:33:16.254092 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.254056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfg9f\" (UniqueName: \"kubernetes.io/projected/1a6cf89c-e10a-4a46-a62f-489113be6ac4-kube-api-access-gfg9f\") pod \"migrator-66df5d496b-l8vbc\" (UID: \"1a6cf89c-e10a-4a46-a62f-489113be6ac4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l8vbc" Oct 31 00:33:16.262601 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.262572 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfg9f\" (UniqueName: \"kubernetes.io/projected/1a6cf89c-e10a-4a46-a62f-489113be6ac4-kube-api-access-gfg9f\") pod \"migrator-66df5d496b-l8vbc\" (UID: \"1a6cf89c-e10a-4a46-a62f-489113be6ac4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l8vbc" Oct 31 00:33:16.286691 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.286653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:16.328693 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.328659 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:16.349981 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.349953 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l8vbc" Oct 31 00:33:16.355339 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.355309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c48c473c-b40e-452f-9e50-5008cddef4b4-signing-key\") pod \"service-ca-5fffc97bdf-f9mbc\" (UID: \"c48c473c-b40e-452f-9e50-5008cddef4b4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" Oct 31 00:33:16.355461 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.355386 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw8cx\" (UniqueName: \"kubernetes.io/projected/c48c473c-b40e-452f-9e50-5008cddef4b4-kube-api-access-rw8cx\") pod \"service-ca-5fffc97bdf-f9mbc\" (UID: \"c48c473c-b40e-452f-9e50-5008cddef4b4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" Oct 31 00:33:16.355461 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.355447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c48c473c-b40e-452f-9e50-5008cddef4b4-signing-cabundle\") pod \"service-ca-5fffc97bdf-f9mbc\" (UID: \"c48c473c-b40e-452f-9e50-5008cddef4b4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" Oct 31 00:33:16.356179 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.356119 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c48c473c-b40e-452f-9e50-5008cddef4b4-signing-cabundle\") pod \"service-ca-5fffc97bdf-f9mbc\" (UID: \"c48c473c-b40e-452f-9e50-5008cddef4b4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" Oct 31 00:33:16.358298 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.358268 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c48c473c-b40e-452f-9e50-5008cddef4b4-signing-key\") pod \"service-ca-5fffc97bdf-f9mbc\" (UID: \"c48c473c-b40e-452f-9e50-5008cddef4b4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" Oct 31 00:33:16.364302 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.364272 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw8cx\" (UniqueName: \"kubernetes.io/projected/c48c473c-b40e-452f-9e50-5008cddef4b4-kube-api-access-rw8cx\") pod \"service-ca-5fffc97bdf-f9mbc\" (UID: \"c48c473c-b40e-452f-9e50-5008cddef4b4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" Oct 31 00:33:16.456736 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.456089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:16.456736 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.456144 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs\") pod \"network-metrics-daemon-2662j\" (UID: \"0d3524d6-1b11-41e9-9862-802d687ea87b\") " pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:16.456736 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:16.456302 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:33:16.456736 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:16.456371 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs podName:0d3524d6-1b11-41e9-9862-802d687ea87b nodeName:}" failed. No retries permitted until 2026-10-31 00:33:48.456350823 +0000 UTC m=+65.177665825 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs") pod "network-metrics-daemon-2662j" (UID: "0d3524d6-1b11-41e9-9862-802d687ea87b") : secret "metrics-daemon-secret" not found Oct 31 00:33:16.459743 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.459595 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:16.478449 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.478414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" Oct 31 00:33:16.558082 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.557772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn99g\" (UniqueName: \"kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g\") pod \"network-check-target-6ftrv\" (UID: \"51aeac6f-2315-42d4-b63e-47efd057cf91\") " pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:16.562935 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.562909 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn99g\" (UniqueName: \"kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g\") pod \"network-check-target-6ftrv\" (UID: \"51aeac6f-2315-42d4-b63e-47efd057cf91\") " pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:16.651909 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.651877 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-65vdk-w4x2d"] Oct 31 00:33:16.655290 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:33:16.655255 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa1579c_25f7_49b4_a9fa_e73dcaeeee35.slice/crio-72cc0ed4af0125f8ef16985f46ed0268c553af7fbc8805414178bf079a98ca2a WatchSource:0}: Error finding container 72cc0ed4af0125f8ef16985f46ed0268c553af7fbc8805414178bf079a98ca2a: Status 404 returned error can't find the container with id 72cc0ed4af0125f8ef16985f46ed0268c553af7fbc8805414178bf079a98ca2a Oct 31 00:33:16.657680 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.657656 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-l8vbc"] Oct 31 00:33:16.658257 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.658166 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:16.658324 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.658260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:16.658439 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:16.658377 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:33:16.658439 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:16.658433 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls podName:bbc352c4-a2e0-496b-8a9e-56b10e016a74 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:17.658413739 +0000 UTC m=+34.379728738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgt89" (UID: "bbc352c4-a2e0-496b-8a9e-56b10e016a74") : secret "insights-runtime-extractor-tls" not found Oct 31 00:33:16.658543 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:16.658484 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:33:16.658543 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:16.658520 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm podName:bbc352c4-a2e0-496b-8a9e-56b10e016a74 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:17.658509576 +0000 UTC m=+34.379824577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgt89" (UID: "bbc352c4-a2e0-496b-8a9e-56b10e016a74") : configmap "kube-rbac-proxy" not found Oct 31 00:33:16.660864 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:33:16.660844 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6cf89c_e10a_4a46_a62f_489113be6ac4.slice/crio-83121842cddf40c16281d4b35d0a25f62152bd2eedb22d543c870beb7d556ecc WatchSource:0}: Error finding container 83121842cddf40c16281d4b35d0a25f62152bd2eedb22d543c870beb7d556ecc: Status 404 returned error can't find the container with id 83121842cddf40c16281d4b35d0a25f62152bd2eedb22d543c870beb7d556ecc Oct 31 00:33:16.706366 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.706285 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:16.710405 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.710379 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:16.797492 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.794293 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-f9mbc"] Oct 31 00:33:16.991741 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:16.991667 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:17.000944 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:17.000857 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6ftrv"] Oct 31 00:33:17.005223 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:33:17.005196 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51aeac6f_2315_42d4_b63e_47efd057cf91.slice/crio-e0840b7fc38d86646dc889012b46f7e6318c88e5c5029dd62798e5b9fc208d45 WatchSource:0}: Error finding container e0840b7fc38d86646dc889012b46f7e6318c88e5c5029dd62798e5b9fc208d45: Status 404 returned error can't find the container with id e0840b7fc38d86646dc889012b46f7e6318c88e5c5029dd62798e5b9fc208d45 Oct 31 00:33:17.020176 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:17.020145 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7tgpr"] Oct 31 00:33:17.023687 ip-10-0-143-188 kubenswrapper[2579]: W1031 00:33:17.023646 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd93b08_b857_4571_a1b5_5eaca4456f8b.slice/crio-a5f8b57af621b627ef137624f5b06149995f706a4eaab762b9593d1dab152f0c WatchSource:0}: Error finding container a5f8b57af621b627ef137624f5b06149995f706a4eaab762b9593d1dab152f0c: Status 404 returned error can't find the container with id a5f8b57af621b627ef137624f5b06149995f706a4eaab762b9593d1dab152f0c Oct 31 00:33:17.111036 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:17.110998 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7tgpr" event={"ID":"ffd93b08-b857-4571-a1b5-5eaca4456f8b","Type":"ContainerStarted","Data":"a5f8b57af621b627ef137624f5b06149995f706a4eaab762b9593d1dab152f0c"} Oct 31 00:33:17.112202 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:17.112177 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" event={"ID":"c48c473c-b40e-452f-9e50-5008cddef4b4","Type":"ContainerStarted","Data":"b460227d1567feb0c8308159d3d1155636bc75461a78280c01dc7c836da8e5af"} Oct 31 00:33:17.113255 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:17.113233 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l8vbc" event={"ID":"1a6cf89c-e10a-4a46-a62f-489113be6ac4","Type":"ContainerStarted","Data":"83121842cddf40c16281d4b35d0a25f62152bd2eedb22d543c870beb7d556ecc"} Oct 31 00:33:17.114322 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:17.114301 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-65vdk-w4x2d" event={"ID":"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35","Type":"ContainerStarted","Data":"72cc0ed4af0125f8ef16985f46ed0268c553af7fbc8805414178bf079a98ca2a"} Oct 31 00:33:17.115351 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:17.115327 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6ftrv" event={"ID":"51aeac6f-2315-42d4-b63e-47efd057cf91","Type":"ContainerStarted","Data":"e0840b7fc38d86646dc889012b46f7e6318c88e5c5029dd62798e5b9fc208d45"} Oct 31 00:33:17.665968 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:17.665876 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:17.665968 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:17.665958 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:17.666489 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:17.666111 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:33:17.666489 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:17.666180 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm podName:bbc352c4-a2e0-496b-8a9e-56b10e016a74 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:19.666160393 +0000 UTC m=+36.387475381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgt89" (UID: "bbc352c4-a2e0-496b-8a9e-56b10e016a74") : configmap "kube-rbac-proxy" not found Oct 31 00:33:17.666623 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:17.666599 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:33:17.666763 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:17.666682 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls podName:bbc352c4-a2e0-496b-8a9e-56b10e016a74 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:19.666667338 +0000 UTC m=+36.387982328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgt89" (UID: "bbc352c4-a2e0-496b-8a9e-56b10e016a74") : secret "insights-runtime-extractor-tls" not found Oct 31 00:33:17.868880 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:17.868832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:17.869163 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:17.869035 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:33:17.869163 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:17.869095 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls podName:330ec43b-421e-42ee-b61e-e26743f9fbb7 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:21.869079147 +0000 UTC m=+38.590394150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls") pod "dns-default-72zjc" (UID: "330ec43b-421e-42ee-b61e-e26743f9fbb7") : secret "dns-default-metrics-tls" not found Oct 31 00:33:19.191528 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:19.191365 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:19.682403 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:19.682365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:19.682593 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:19.682434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:19.682593 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:19.682574 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:33:19.682739 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:19.682653 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm podName:bbc352c4-a2e0-496b-8a9e-56b10e016a74 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:23.682618014 +0000 UTC m=+40.403933002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgt89" (UID: "bbc352c4-a2e0-496b-8a9e-56b10e016a74") : configmap "kube-rbac-proxy" not found Oct 31 00:33:19.683117 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:19.683095 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:33:19.683228 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:19.683151 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls podName:bbc352c4-a2e0-496b-8a9e-56b10e016a74 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:23.683136642 +0000 UTC m=+40.404451627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgt89" (UID: "bbc352c4-a2e0-496b-8a9e-56b10e016a74") : secret "insights-runtime-extractor-tls" not found Oct 31 00:33:20.128915 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:20.128868 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l8vbc" event={"ID":"1a6cf89c-e10a-4a46-a62f-489113be6ac4","Type":"ContainerStarted","Data":"416bf9aa326bb558564a13a3effc4ca0829c9cef3658b1006e3bc06ce45314df"} Oct 31 00:33:20.131845 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:20.131811 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vdxdv" event={"ID":"9f037fa6-1d56-42b5-922e-007f1af8552c","Type":"ContainerStarted","Data":"60f0ff5f72b92f1ea2bf410d5036708a1f83b8ba6374888db322845fe220dd26"} Oct 31 00:33:20.146714 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:20.146652 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l8vbc" podStartSLOduration=2.408601129 podStartE2EDuration="5.146620238s" totalImagesPullingTime="2.738019109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:20.145347209 +0000 UTC m=+36.866662241" watchObservedRunningTime="2026-10-31 00:33:20.146620238 +0000 UTC m=+36.867935247" Oct 31 00:33:20.163061 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:20.162997 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vdxdv" podStartSLOduration=1.433645275 podStartE2EDuration="6.162986827s" totalImagesPullingTime="4.729341552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:20.161944228 +0000 UTC m=+36.883259250" watchObservedRunningTime="2026-10-31 00:33:20.162986827 +0000 UTC m=+36.884301835" Oct 31 00:33:21.137775 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:21.137721 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" event={"ID":"c48c473c-b40e-452f-9e50-5008cddef4b4","Type":"ContainerStarted","Data":"46dee273d0ec75ef620afb058c9569cd280ac9cec9f28df62ac3b683b6c42273"} Oct 31 00:33:21.142514 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:21.141170 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-65vdk-w4x2d" event={"ID":"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35","Type":"ContainerStarted","Data":"f503f1766c023f8399506061e61349cc6e9388be692a0560c0039fe55393c728"} Oct 31 00:33:21.157229 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:21.156968 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" podStartSLOduration=0.95307073 podStartE2EDuration="5.156949579s" totalImagesPullingTime="4.203878849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:21.153846767 +0000 UTC m=+37.875161773" watchObservedRunningTime="2026-10-31 00:33:21.156949579 +0000 UTC m=+37.878264587" Oct 31 00:33:21.583558 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:21.583402 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:21.903766 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:21.903106 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:21.903766 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:21.903304 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:33:21.903766 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:21.903374 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls podName:330ec43b-421e-42ee-b61e-e26743f9fbb7 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:29.903356913 +0000 UTC m=+46.624671911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls") pod "dns-default-72zjc" (UID: "330ec43b-421e-42ee-b61e-e26743f9fbb7") : secret "dns-default-metrics-tls" not found Oct 31 00:33:22.007824 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:22.007786 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:22.146221 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:22.146188 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6ftrv" event={"ID":"51aeac6f-2315-42d4-b63e-47efd057cf91","Type":"ContainerStarted","Data":"4b58b477ddfb3e58f5774edf682fd076bdd5826d1724c635783699ee180604ab"} Oct 31 00:33:22.146704 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:22.146506 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:22.175680 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:22.175562 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-65vdk-w4x2d" podStartSLOduration=2.820692594 podStartE2EDuration="7.175544242s" totalImagesPullingTime="4.354851648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:21.170784487 +0000 UTC m=+37.892099498" watchObservedRunningTime="2026-10-31 00:33:22.175544242 +0000 UTC m=+38.896859252" Oct 31 00:33:22.176728 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:22.176687 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6ftrv" podStartSLOduration=33.224243093 podStartE2EDuration="38.176674853s" totalImagesPullingTime="4.95243176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:22.175237422 +0000 UTC m=+38.896552431" watchObservedRunningTime="2026-10-31 00:33:22.176674853 +0000 UTC m=+38.897989859" Oct 31 00:33:22.181752 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:22.181663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:22.275921 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:22.275827 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:22.390660 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:22.390574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:22.579039 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:22.579006 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:22.952894 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:22.952810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:23.718345 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:23.718310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:23.718760 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:23.718370 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:23.718760 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:23.718453 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:33:23.718760 ip-10-0-143-188 kubenswrapper[2579]: E1031 00:33:23.718526 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls podName:bbc352c4-a2e0-496b-8a9e-56b10e016a74 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:31.718506868 +0000 UTC m=+48.439821853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgt89" (UID: "bbc352c4-a2e0-496b-8a9e-56b10e016a74") : secret "insights-runtime-extractor-tls" not found Oct 31 00:33:23.719065 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:23.719043 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:23.745132 ip-10-0-143-188 kubenswrapper[2579]: I1031 00:33:23.745094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:23.883099 ip-10-0-143-188 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 00:33:23.900062 ip-10-0-143-188 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 00:33:23.900231 ip-10-0-143-188 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 00:33:23.900283 ip-10-0-143-188 systemd[1]: kubelet.service: Consumed 2.446s CPU time. Oct 31 00:33:23.919159 ip-10-0-143-188 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:33:24.015583 ip-10-0-143-188 kubenswrapper[4519]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:33:24.015583 ip-10-0-143-188 kubenswrapper[4519]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:33:24.015583 ip-10-0-143-188 kubenswrapper[4519]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:33:24.015583 ip-10-0-143-188 kubenswrapper[4519]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:33:24.017801 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017707 4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:33:24.017801 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017800 4519 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:33:24.017801 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017804 4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017809 4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017812 4519 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017815 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017818 4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017821 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017824 4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017827 4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017830 4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017833 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017835 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017838 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017840 4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017843 4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017846 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017848 4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017851 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017854 4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017856 4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:33:24.017916 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017859 4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017862 4519 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017865 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017867 4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017875 4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017878 4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017881 4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017884 4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017887 4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017890 4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017893 4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017896 4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017899 4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017901 4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017905 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017908 4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017910 4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017913 4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017916 4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017919 4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:33:24.018394 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017921 4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017924 4519 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017927 4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017930 4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017933 4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017935 4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017938 4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017940 4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017943 4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017946 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017948 4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017951 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017954 4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017956 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017959 4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017962 4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017964 4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017969 4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017972 4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017974 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:33:24.018918 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017977 4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017979 4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017982 4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017984 4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017987 4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017990 4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017992 4519 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017995 4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.017998 4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018000 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018003 4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018005 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018007 4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018010 4519 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018013 4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018015 4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018018 4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018020 4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018023 4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018025 4519 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:33:24.019459 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018028 4519 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018030 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018033 4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018036 4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018039 4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018041 4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018044 4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018046 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018049 4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018052 4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018056 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018059 4519 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018061 4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018064 4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018068 4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018072 4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018075 4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018078 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018080 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:33:24.019989 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018083 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018086 4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018088 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018093 4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018097 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018100 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018914 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018923 4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018926 4519 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018929 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018933 4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018936 4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018939 4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018942 4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018944 4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018947 4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018950 4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018952 4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018955 4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:33:24.020464 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018958 4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018961 4519 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018963 4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018967 4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018969 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018973 4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018977 4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018981 4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018984 4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018987 4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018991 4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018993 4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018996 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.018999 4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019002 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019004 4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019007 4519 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019009 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019012 4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:33:24.020969 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019014 4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019017 4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019019 4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019023 4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019026 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019028 4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019031 4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019033 4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019036 4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019038 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019041 4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019044 4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019046 4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019048 4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019051 4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019054 4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019057 4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019060 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019063 4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:33:24.021449 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019066 4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019068 4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019071 4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019074 4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019077 4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019079 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019082 4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019084 4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019087 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019090 4519 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019092 4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019095 4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019098 4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019101 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019103 4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019106 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019109 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019111 4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019114 4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019117 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:33:24.021963 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019119 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019122 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019124 4519 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019127 4519 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019129 4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019132 4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019134 4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019138 4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019140 4519 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019143 4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019145 4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019149 4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019151 4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019155 4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019157 4519 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019160 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019163 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019168 4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019172 4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019175 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:33:24.022472 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019178 4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019181 4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019184 4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019186 4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019189 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019191 4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019194 4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019196 4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019199 4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019201 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019204 4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019207 4519 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019209 4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019212 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019216 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019777 4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019784 4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019787 4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019791 4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:33:24.022996 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019794 4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019797 4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019799 4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019802 4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019805 4519 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019808 4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019811 4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019814 4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019817 4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019820 4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019822 4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019825 4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019828 4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019830 4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019833 4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019836 4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019838 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019840 4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019843 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:33:24.023476 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019845 4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019848 4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019850 4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019853 4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019856 4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019859 4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019861 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019864 4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019866 4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019869 4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019871 4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019874 4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019880 4519 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019883 4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019886 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019889 4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019891 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019894 4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019897 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:33:24.024205 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019899 4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019902 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019905 4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019908 4519 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019910 4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019913 4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019915 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019918 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019921 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019923 4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019926 4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019928 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019931 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019934 4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019936 4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019939 4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019942 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019944 4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019947 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:33:24.025010 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019950 4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019953 4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019955 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019957 4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019960 4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019962 4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019967 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019970 4519 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019973 4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019975 4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019978 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019981 4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019984 4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019986 4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019989 4519 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019992 4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019995 4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.019998 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020000 4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020003 4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:33:24.025761 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020006 4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020009 4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020011 4519 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020014 4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020016 4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020019 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020022 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020024 4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020027 4519 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020031 4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020036 4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020040 4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020044 4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020048 4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020051 4519 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020054 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020057 4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020059 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020064 4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:33:24.026941 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020067 4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020069 4519 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020072 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020076 4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020079 4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020081 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020173 4519 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020181 4519 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020188 4519 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020194 4519 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020198 4519 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020201 4519 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020206 4519 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020211 4519 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020214 4519 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020217 4519 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020221 4519 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020224 4519 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020227 4519 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020230 4519 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020233 4519 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020236 4519 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020239 4519 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:33:24.027797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020242 4519 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020247 4519 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020250 4519 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020253 4519 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020257 4519 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020260 4519 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020265 4519 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020268 4519 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020272 4519 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020276 4519 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020279 4519 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020282 4519 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020301 4519 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020306 4519 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020309 4519 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020314 4519 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020318 4519 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020321 4519 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020325 4519 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020328 4519 flags.go:64] FLAG: --enable-server="true" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020331 4519 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020336 4519 flags.go:64] FLAG: --event-burst="100" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020339 4519 flags.go:64] FLAG: --event-qps="50" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020342 4519 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:33:24.028991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020345 4519 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020348 4519 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020353 4519 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020356 4519 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020359 4519 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020362 4519 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020365 4519 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020368 4519 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020372 4519 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020375 4519 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020378 4519 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020381 4519 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020384 4519 flags.go:64] FLAG: --feature-gates="" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020388 4519 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020392 4519 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020397 4519 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020403 4519 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020406 4519 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020409 4519 flags.go:64] FLAG: --help="false" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020412 4519 flags.go:64] FLAG: --hostname-override="ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020416 4519 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020419 4519 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020422 4519 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:33:24.030113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020425 4519 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020429 4519 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020432 4519 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020435 4519 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020438 4519 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020442 4519 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020445 4519 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020449 4519 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020452 4519 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020455 4519 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020458 4519 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020461 4519 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020463 4519 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020467 4519 flags.go:64] FLAG: --lock-file="" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020469 4519 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020473 4519 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020476 4519 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020481 4519 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020484 4519 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020487 4519 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020490 4519 flags.go:64] FLAG: --logging-format="text" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020493 4519 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020496 4519 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020499 4519 flags.go:64] FLAG: --manifest-url="" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020504 4519 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:33:24.031142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020511 4519 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020514 4519 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020518 4519 flags.go:64] FLAG: --max-pods="110" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020522 4519 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020525 4519 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020528 4519 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020531 4519 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020534 4519 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020537 4519 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020540 4519 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020547 4519 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020550 4519 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020554 4519 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020557 4519 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020560 4519 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020563 4519 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020567 4519 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020570 4519 flags.go:64] FLAG: --port="10250" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020573 4519 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020576 4519 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0c0c4713797f47505" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020579 4519 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020582 4519 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020585 4519 flags.go:64] FLAG: --register-node="true" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020588 4519 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020592 4519 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020595 4519 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:33:24.032216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020598 4519 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020601 4519 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020605 4519 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020608 4519 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020611 4519 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020616 4519 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020621 4519 flags.go:64] FLAG: --runonce="false" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020624 4519 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020627 4519 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020644 4519 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020648 4519 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020651 4519 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020654 4519 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020657 4519 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020661 4519 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020664 4519 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020667 4519 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020670 4519 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020673 4519 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020676 4519 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020679 4519 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020683 4519 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020688 4519 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020691 4519 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020694 4519 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:33:24.033335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020699 4519 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020702 4519 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020704 4519 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020708 4519 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020711 4519 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020714 4519 flags.go:64] FLAG: --v="2" Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020718 4519 flags.go:64] FLAG: --version="false" Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020722 4519 flags.go:64] FLAG: --vmodule="" Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020726 4519 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.020730 4519 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020866 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020869 4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020874 4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020878 4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020881 4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020884 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020886 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020889 4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020892 4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020894 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020897 4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:33:24.034484 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020900 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020902 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020905 4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020907 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020910 4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020913 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020915 4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020918 4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020921 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020924 4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020927 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020929 4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020933 4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020938 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020941 4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020944 4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020947 4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020949 4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020952 4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:33:24.035099 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020955 4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020957 4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020960 4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020962 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020967 4519 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020971 4519 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020974 4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020976 4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020979 4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020982 4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020985 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020988 4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020991 4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020993 4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020996 4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.020998 4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021001 4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021004 4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021007 4519 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021009 4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:33:24.035644 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021012 4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021015 4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021019 4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021023 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021026 4519 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021029 4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021032 4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021034 4519 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021037 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021040 4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021043 4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021045 4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021048 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021051 4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021053 4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021056 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021059 4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021064 4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021066 4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:33:24.036206 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021069 4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021072 4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021074 4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021077 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021080 4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021082 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021085 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021088 4519 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021091 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021093 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021096 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021099 4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021101 4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021104 4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021106 4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021109 4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021112 4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021114 4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021117 4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:33:24.036754 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021119 4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:33:24.037224 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021122 4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:33:24.037224 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021124 4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:33:24.037224 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021127 4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:33:24.037224 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021130 4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:33:24.037224 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021132 4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:33:24.037224 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021135 4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:33:24.037224 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021137 4519 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:33:24.037224 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021140 4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:33:24.037224 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021143 4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:33:24.037224 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021147 4519 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:33:24.037224 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021152 4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:33:24.037224 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021154 4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:33:24.037224 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021157 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:33:24.037224 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021160 4519 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:33:24.037224 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021163 4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:33:24.037224 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021166 4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:33:24.037224 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021168 4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:33:24.037750 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.021195 4519 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:33:24.037750 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021234 4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:33:24.037750 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021239 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:33:24.037750 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021243 4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:33:24.037750 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021247 4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:33:24.037750 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021250 4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:33:24.037750 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021253 4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:33:24.037750 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021256 4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:33:24.037750 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021258 4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:33:24.037750 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021261 4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:33:24.037750 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021264 4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:33:24.037750 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021267 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:33:24.037750 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021270 4519 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:33:24.037750 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021272 4519 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:33:24.037750 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021275 4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:33:24.037750 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021278 4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021280 4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021283 4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021285 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021288 4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021291 4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021293 4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021296 4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021298 4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021303 4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021306 4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021310 4519 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021313 4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021315 4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021318 4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021322 4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021326 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021329 4519 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021332 4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:33:24.038295 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021335 4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021338 4519 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021341 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021344 4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021347 4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021350 4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021352 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021355 4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021357 4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021360 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021363 4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021366 4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021369 4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021372 4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021374 4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021377 4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021379 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021382 4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021384 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:33:24.038810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021388 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021390 4519 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021393 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021397 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021399 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021403 4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021406 4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021408 4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021411 4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021413 4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021416 4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021419 4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021421 4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021424 4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021426 4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021429 4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021431 4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021434 4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021436 4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021439 4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:33:24.039292 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021441 4519 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021444 4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021446 4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021449 4519 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021457 4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021460 4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021463 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021466 4519 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021468 4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021471 4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021474 4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021476 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021479 4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021481 4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021484 4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021488 4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021490 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021494 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021496 4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021499 4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:33:24.039814 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021501 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021504 4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021507 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021509 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021512 4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021514 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021517 4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021520 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021522 4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021525 4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021527 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021530 4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021532 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.021535 4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.027618 4519 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.027660 4519 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.027687 4519 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.027695 4519 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027789 4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027798 4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:33:24.040299 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027803 4519 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027807 4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027826 4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027832 4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027837 4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027842 4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027846 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027851 4519 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027855 4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027860 4519 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027864 4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027868 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027873 4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027877 4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027883 4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027888 4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027893 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027898 4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027902 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027907 4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:33:24.040813 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027912 4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027916 4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027921 4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027925 4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027930 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027935 4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027940 4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027945 4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027949 4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027954 4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027959 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027963 4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027967 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027972 4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027976 4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027981 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027987 4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027991 4519 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.027996 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028000 4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:33:24.041297 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028004 4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028009 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028013 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028018 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028023 4519 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028028 4519 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028033 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028037 4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028041 4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028046 4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028050 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028055 4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028059 4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028064 4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028068 4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028073 4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028077 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028081 4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028086 4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028090 4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:33:24.041812 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028095 4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028099 4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028106 4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028113 4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028118 4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028123 4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028128 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028133 4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028137 4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028141 4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028145 4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028151 4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028155 4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028160 4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028164 4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028169 4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028173 4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028179 4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028183 4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:33:24.042301 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028188 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028192 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028196 4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028200 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028205 4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028209 4519 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028213 4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028217 4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028222 4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028227 4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028231 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028235 4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028242 4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028250 4519 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028255 4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028260 4519 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028264 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028269 4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028273 4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:33:24.042888 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028278 4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:33:24.043359 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028282 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:33:24.043359 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028287 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:33:24.043359 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028291 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:33:24.043359 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028295 4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:33:24.043359 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028300 4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:33:24.043359 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.028348 4519 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:33:24.043359 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028583 4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:33:24.043359 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028592 4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:33:24.043359 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028597 4519 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:33:24.043359 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028602 4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:33:24.043359 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028606 4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:33:24.043359 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028613 4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:33:24.043359 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028617 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:33:24.043359 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028622 4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:33:24.043359 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028626 4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028651 4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028656 4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028660 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028664 4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028668 4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028672 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028676 4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028680 4519 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028684 4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028689 4519 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028693 4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028697 4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028701 4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028705 4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028710 4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028715 4519 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028718 4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028722 4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028726 4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:33:24.043749 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028730 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028734 4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028739 4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028743 4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028748 4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028752 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028757 4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028761 4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028765 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028770 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028777 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028781 4519 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028786 4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028791 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028795 4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028800 4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028804 4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028808 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028813 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:33:24.044279 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028817 4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028822 4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028826 4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028832 4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028836 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028841 4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028847 4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028853 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028858 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028863 4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028867 4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028872 4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028877 4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028881 4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028886 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028890 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028894 4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028899 4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028903 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:33:24.044764 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028908 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028912 4519 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028917 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028921 4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028925 4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028931 4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028936 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028944 4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028948 4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028953 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028957 4519 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028964 4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028970 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028976 4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028981 4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028985 4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028989 4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028994 4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.028998 4519 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:33:24.045226 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029002 4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029006 4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029010 4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029014 4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029018 4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029022 4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029026 4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029030 4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029035 4519 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029039 4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029044 4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029048 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029053 4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029057 4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029061 4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029066 4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029070 4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029074 4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029079 4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029085 4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:33:24.045924 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029091 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:33:24.046821 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.029139 4519 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:33:24.046821 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029195 4519 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:33:24.046821 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029203 4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:33:24.046821 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029209 4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:33:24.046821 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029213 4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:33:24.046821 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029218 4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:33:24.046821 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029222 4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:33:24.046821 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029227 4519 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:33:24.046821 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029232 4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:33:24.046821 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029236 4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:33:24.046821 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029241 4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:33:24.046821 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029245 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:33:24.046821 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029250 4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:33:24.046821 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029254 4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:33:24.046821 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029258 4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:33:24.046821 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029262 4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029267 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029271 4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029276 4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029280 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029285 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029290 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029294 4519 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029299 4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029303 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029307 4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029311 4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029316 4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029320 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029325 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029329 4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029334 4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029344 4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029348 4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:33:24.047250 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029353 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029357 4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029363 4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029370 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029377 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029382 4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029386 4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029390 4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029395 4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029399 4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029403 4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029408 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029412 4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029417 4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029421 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029425 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029430 4519 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029434 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029440 4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:33:24.047746 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029444 4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029449 4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029453 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029457 4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029462 4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029466 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029470 4519 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029477 4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029482 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029487 4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029491 4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029500 4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029504 4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029509 4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029513 4519 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029518 4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029522 4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029527 4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029531 4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:33:24.048242 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029536 4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029540 4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029544 4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029549 4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029553 4519 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029558 4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029562 4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029566 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029571 4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029575 4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029580 4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029584 4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029588 4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029593 4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029597 4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029601 4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029606 4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029610 4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029614 4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029619 4519 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:33:24.048711 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029623 4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029627 4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029649 4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029654 4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029661 4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029666 4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029683 4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029688 4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029693 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029698 4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029702 4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029706 4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029711 4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029715 4519 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:24.029722 4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.029822 4519 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.031440 4519 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.031522 4519 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.031825 4519 server.go:1014] "Starting client certificate rotation" Oct 31 00:33:24.049247 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.031945 4519 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:33:24.049743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.032003 4519 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:27:44 +0000 UTC" deadline="2028-06-07 11:13:39.274580513 +0000 UTC" Oct 31 00:33:24.049743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.032032 4519 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14050h40m15.242550689s" Oct 31 00:33:24.049743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.032596 4519 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:33:24.049743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.034358 4519 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:33:24.049743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.035870 4519 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:33:24.049743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.037389 4519 log.go:25] "Validated CRI v1 image API" Oct 31 00:33:24.049743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.038190 4519 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:33:24.049743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.039742 4519 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 862cc37e-30b9-42ec-831f-b1b4f8ed4b93:/dev/nvme0n1p3 8a805194-6aa5-477c-a701-efd66da57918:/dev/nvme0n1p4] Oct 31 00:33:24.049995 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.039764 4519 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/045eb8964330e018a313575e35948e73f5c1886605fbc9eff322cbe746276a31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/045eb8964330e018a313575e35948e73f5c1886605fbc9eff322cbe746276a31/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/104f041ebdf17becbae4fd9723914460a4894bee6fe87894a6733eaca1791f07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/104f041ebdf17becbae4fd9723914460a4894bee6fe87894a6733eaca1791f07/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2367e34dfbf292d0fff1e7b7cec1f944113d6acd7f49a860a725b7404cc10514/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2367e34dfbf292d0fff1e7b7cec1f944113d6acd7f49a860a725b7404cc10514/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53071f197595f817a55460bb672281104ec03035e16832b108638a3111746883/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53071f197595f817a55460bb672281104ec03035e16832b108638a3111746883/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70578d806cf234fed0c27479459e0a55693f200e9bafce2e118537e5500d9156/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70578d806cf234fed0c27479459e0a55693f200e9bafce2e118537e5500d9156/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72cc0ed4af0125f8ef16985f46ed0268c553af7fbc8805414178bf079a98ca2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72cc0ed4af0125f8ef16985f46ed0268c553af7fbc8805414178bf079a98ca2a/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/749ca676ba095a8ba9698d1d2464f22a9935fbbbd185907bd44c09179abfde60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/749ca676ba095a8ba9698d1d2464f22a9935fbbbd185907bd44c09179abfde60/userdata/shm major:0 minor:57 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83121842cddf40c16281d4b35d0a25f62152bd2eedb22d543c870beb7d556ecc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83121842cddf40c16281d4b35d0a25f62152bd2eedb22d543c870beb7d556ecc/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a258ac5e07beeb38e563de3d745b46806e9edd50d635777975622c06721a478/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a258ac5e07beeb38e563de3d745b46806e9edd50d635777975622c06721a478/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a311b2965f9d8ee83128a34f312fd8708da25842aef8c59e4e7cd24ea93eed99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a311b2965f9d8ee83128a34f312fd8708da25842aef8c59e4e7cd24ea93eed99/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5f8b57af621b627ef137624f5b06149995f706a4eaab762b9593d1dab152f0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5f8b57af621b627ef137624f5b06149995f706a4eaab762b9593d1dab152f0c/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b460227d1567feb0c8308159d3d1155636bc75461a78280c01dc7c836da8e5af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b460227d1567feb0c8308159d3d1155636bc75461a78280c01dc7c836da8e5af/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5d15af0f24c23067306c0897776d845bd443e7c7ac469369e02d09029476d44/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5d15af0f24c23067306c0897776d845bd443e7c7ac469369e02d09029476d44/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df08eacd54bfadb4d947b1cc2f064e42c083f6a0048b4612fa5539ca9985b4f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df08eacd54bfadb4d947b1cc2f064e42c083f6a0048b4612fa5539ca9985b4f6/userdata/shm major:0 minor:66 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0840b7fc38d86646dc889012b46f7e6318c88e5c5029dd62798e5b9fc208d45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0840b7fc38d86646dc889012b46f7e6318c88e5c5029dd62798e5b9fc208d45/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2d4327725d3a33a287e14e7d623f858244d368269196a373031c49862b08fde/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2d4327725d3a33a287e14e7d623f858244d368269196a373031c49862b08fde/userdata/shm major:0 minor:58 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d3524d6-1b11-41e9-9862-802d687ea87b/volumes/kubernetes.io~projected/kube-api-access-q9pf2:{mountpoint:/var/lib/kubelet/pods/0d3524d6-1b11-41e9-9862-802d687ea87b/volumes/kubernetes.io~projected/kube-api-access-q9pf2 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03/volumes/kubernetes.io~projected/kube-api-access-2h6qd:{mountpoint:/var/lib/kubelet/pods/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03/volumes/kubernetes.io~projected/kube-api-access-2h6qd major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a6cf89c-e10a-4a46-a62f-489113be6ac4/volumes/kubernetes.io~projected/kube-api-access-gfg9f:{mountpoint:/var/lib/kubelet/pods/1a6cf89c-e10a-4a46-a62f-489113be6ac4/volumes/kubernetes.io~projected/kube-api-access-gfg9f major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fe15ece-2b2c-4a7d-956f-950a1829b2ac/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1fe15ece-2b2c-4a7d-956f-950a1829b2ac/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fe15ece-2b2c-4a7d-956f-950a1829b2ac/volumes/kubernetes.io~projected/kube-api-access-8gvtc:{mountpoint:/var/lib/kubelet/pods/1fe15ece-2b2c-4a7d-956f-950a1829b2ac/volumes/kubernetes.io~projected/kube-api-access-8gvtc major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fe15ece-2b2c-4a7d-956f-950a1829b2ac/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1fe15ece-2b2c-4a7d-956f-950a1829b2ac/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21754645-32d8-4b1b-a7f5-00594f5854f3/volumes/kubernetes.io~projected/kube-api-access-6n24w:{mountpoint:/var/lib/kubelet/pods/21754645-32d8-4b1b-a7f5-00594f5854f3/volumes/kubernetes.io~projected/kube-api-access-6n24w major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/330ec43b-421e-42ee-b61e-e26743f9fbb7/volumes/kubernetes.io~projected/kube-api-access-sm9r6:{mountpoint:/var/lib/kubelet/pods/330ec43b-421e-42ee-b61e-e26743f9fbb7/volumes/kubernetes.io~projected/kube-api-access-sm9r6 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3346ce41-dd23-4e63-8944-43f2d7d4aaec/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3346ce41-dd23-4e63-8944-43f2d7d4aaec/volumes/kubernetes.io~secret/agent-certs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51aeac6f-2315-42d4-b63e-47efd057cf91/volumes/kubernetes.io~projected/kube-api-access-dn99g:{mountpoint:/var/lib/kubelet/pods/51aeac6f-2315-42d4-b63e-47efd057cf91/volumes/kubernetes.io~projected/kube-api-access-dn99g major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/560532f2-4a95-4180-a772-b269efd3d73d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/560532f2-4a95-4180-a772-b269efd3d73d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/560532f2-4a95-4180-a772-b269efd3d73d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/560532f2-4a95-4180-a772-b269efd3d73d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/560532f2-4a95-4180-a772-b269efd3d73d/volumes/kubernetes.io~projected/kube-api-access-h9tcp:{mountpoint:/var/lib/kubelet/pods/560532f2-4a95-4180-a772-b269efd3d73d/volumes/kubernetes.io~projected/kube-api-access-h9tcp major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64dd929e-8d9d-47f3-bcb9-a0371d0fc279/volumes/kubernetes.io~projected/kube-api-access-xpnf2:{mountpoint:/var/lib/kubelet/pods/64dd929e-8d9d-47f3-bcb9-a0371d0fc279/volumes/kubernetes.io~projected/kube-api-access-xpnf2 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91f27c45-482c-4a05-998e-18b57ebfa499/volumes/kubernetes.io~projected/kube-api-access-rblj9:{mountpoint:/var/lib/kubelet/pods/91f27c45-482c-4a05-998e-18b57ebfa499/volumes/kubernetes.io~projected/kube-api-access-rblj9 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/993fd395-dba5-4c79-99d3-72792b8c273c/volumes/kubernetes.io~projected/kube-api-access-zwqpz:{mountpoint:/var/lib/kubelet/pods/993fd395-dba5-4c79-99d3-72792b8c273c/volumes/kubernetes.io~projected/kube-api-access-zwqpz major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f037fa6-1d56-42b5-922e-007f1af8552c/volumes/kubernetes.io~projected/kube-api-access-ssv6w:{mountpoint:/var/lib/kubelet/pods/9f037fa6-1d56-42b5-922e-007f1af8552c/volumes/kubernetes.io~projected/kube-api-access-ssv6w major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f037fa6-1d56-42b5-922e-007f1af8552c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9f037fa6-1d56-42b5-922e-007f1af8552c/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35/volumes/kubernetes.io~projected/kube-api-access-8qcz5:{mountpoint:/var/lib/kubelet/pods/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35/volumes/kubernetes.io~projected/kube-api-access-8qcz5 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbc352c4-a2e0-496b-8a9e-56b10e016a74/volumes/kubernetes.io~projected/kube-api-access-45lrv:{mountpoint:/var/lib/kubelet/pods/bbc352c4-a2e0-496b-8a9e-56b10e016a74/volumes/kubernetes.io~projected/kube-api-access-45lrv major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c48c473c-b40e-452f-9e50-5008cddef4b4/volumes/kubernetes.io~projected/kube-api-access-rw8cx:{mountpoint:/var/lib/kubelet/pods/c48c473c-b40e-452f-9e50-5008cddef4b4/volumes/kubernetes.io~projected/kube-api-access-rw8cx major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c48c473c-b40e-452f-9e50-5008cddef4b4/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/c48c473c-b40e-452f-9e50-5008cddef4b4/volumes/kubernetes.io~secret/signing-key major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de6e63f3-65e8-4d4d-bad4-015fed1391a3/volumes/kubernetes.io~projected/kube-api-access-qfw88:{mountpoint:/var/lib/kubelet/pods/de6e63f3-65e8-4d4d-bad4-015fed1391a3/volumes/kubernetes.io~projected/kube-api-access-qfw88 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffd93b08-b857-4571-a1b5-5eaca4456f8b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ffd93b08-b857-4571-a1b5-5eaca4456f8b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:215 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/d05cbb63056e3f87a1b0e15aea982ee1893a75824f3754a02ae2e87edfb92890/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/dee9ef759b549df95d7a8db460b64cde7ea9ad6e63367674a5889295ba1d7b1b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/357c65e95b6494b4edc4f2b0b26ddf1f0b9979b37c849fa53e674229a4244892/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/98530b3d59b19352d7f75d4de2a27d7943ce92ecf4229abd4da9976ad94c2ad0/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6037efd4eb6270f0009a0bdbd41f2e0da1117183faf9246c310bc5450a0223d4/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/dbe0dcd4d8fab2109943f789de76aba932d4a75c6c33fb720c8cb1697a42331c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/17cd9affbe0f26a914d875f01093ec98fa7a383b4f3008a8fd6d05993b14243b/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/fdcd2e1187709978be15df0fd4779efab6b743a264333d9b1cdcdbc522e56f8d/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e4e14fea1d6b7e1266581cb22c2f82bc81c5e862ccff7fdb3c3d535c0b2a86a8/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/4e5466046490edb7f988b9f308e39851d1f17559b022ab738f75265a7b872555/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/3565785bb21f42827d88b83171b4fa87432f9efa28769e750f5916239dc9b1a3/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/25a8072dd35154ce421da8ee9063ce69196251bc687ee529949df955f1609987/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0df1ba640158f46a7f907da6d9f9e4d2fbbd2e00b3a6fbd9c6b34aa10dae9e10/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/ada3a5ab0205f5ff875b00a911966456cf8390d9bb7e9cb0912aa3efc9a18f62/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/94ed080ef4f244e69d1334c1d8ee8c1e262aa67c11de781bad797fc4c5c57624/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/7d50a70680a3bfc2a69d137a0b42e264552282f8082741ae2625bbd0335a796d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b7eab353ab91e6e3d8ef745b22536bd1a6903d3e49427b0a99d6d11cdec6da84/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/054a5a2581c53c17dcaebefea4f067d961fa1de00c48441f23cbf989828aba8d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f9a7c97f7a25763b691f02205b62d96aab45ebd0d37e5d5a4ccbd134d240a82e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0546a6aced14ea6466fd4eba349bc7ff2afacceac9231e8fed6a8f50edba7472/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/28dd982535b6d5a3c034b3ad7caa1a0a26dac47eeb61fe621a41be94c35ffbd5/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/38d464c1c007b09c9cacf8e060fce682f325d980d608a34f6b9ec781e6d34adf/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/3a7427fb91ce22987c06424ce554d622cd0215596cf037140e2a75cd292d6565/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/6bccaed0176ad76c7dab56b7c17fbfdd287dc2f6b85509a32138c9544c82dfd3/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/d0c4199103cb3096f5f3cf1ee16c58c611b50018276f8f035dbd68db2f2a6d7f/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/ecfda14de1e86bda36194337f6742576ce74c3d1925505fe36dbf2aa1a0e81ce/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/0069fa8b55ad6817b078e17cc15d0ca4a0be84ca5fa0a96eff6f42b8c1918623/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/f9c18482d3ad2bbd3311ee4d76c8a53105ca2532c90f486e12d21b160ab5d18c/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/91d95cd3199184619e80cf55a256b46da971e077d13dec9541365928132b2ab1/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/b20cebab042e5652e3a2c949d1447f9185c0c4f101166217e94a39deaca87f31/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/00bec92fe68438f47c3b9b9c6aedb749e888d3b921d3db4aff9191b0c023eb43/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/e8c4396958b8a45ba616a1ed0653228b405d4ab5c809314b4ac58d92e0ed4d65/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/a2c99c267b775b37e017caca45de37a5bb1f2cc2338d3396ef2a81da11bf28a5/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/881d87f3ab04146eed8f3c3a061f85d57937799b44f38ea3025e9c679d3dba7f/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/1dcb0fd910bd05f5c27a9e27dbd0010db4e1542c8a9306937d1bf5ce8787ab9d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/054f5b000d7d665c0d526225540de6954cdd40ba0c17b5701bbdd73295aa3701/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/7436ec2f167186f662adcfe226dab002e038228f255b4e5fa4028d5716e328cf/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/9446b0a203dff5bcef7f5c0f6df5fc4c4d930ca36bcfc521cf59012568a11e4d/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/0b0cee8c3691507e9215989c6c19a0af7da7fe2a136a9d9646a290899620458b/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/4b618f36914c3f21ba7706e2faea0f591bebdb3e690de53875358c294695cccf/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/6cd19d5e97c63e3352fb2569cc4ccd5c76cbbda9d28a8921e1e1d0d25ae45fa1/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/644e74e8fcc20b3d8e4d2ec19e26e674efe78ccc5ce9d792a6e4acad382c6cb8/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/ab75d5c31fda6cff32bfee3223f466383825d907a1ed43ab3ef7e8564eaaf03a/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.047565 4519 manager.go:217] Machine: {Timestamp:2026-10-31 00:33:24.047032972 +0000 UTC m=+0.076213353 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101413 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec230cfd53d2a7adfdada89d18b0a3d4 SystemUUID:ec230cfd-53d2-a7ad-fdad-a89d18b0a3d4 BootID:c53db8fc-955b-4ecc-8c05-9a0a74ae7c16 Filesystems:[{Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f037fa6-1d56-42b5-922e-007f1af8552c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9f037fa6-1d56-42b5-922e-007f1af8552c/volumes/kubernetes.io~projected/kube-api-access-ssv6w DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1fe15ece-2b2c-4a7d-956f-950a1829b2ac/volumes/kubernetes.io~projected/kube-api-access-8gvtc DeviceMajor:0 DeviceMinor:50 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffd93b08-b857-4571-a1b5-5eaca4456f8b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0840b7fc38d86646dc889012b46f7e6318c88e5c5029dd62798e5b9fc208d45/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2d4327725d3a33a287e14e7d623f858244d368269196a373031c49862b08fde/userdata/shm DeviceMajor:0 DeviceMinor:58 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c48c473c-b40e-452f-9e50-5008cddef4b4/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03/volumes/kubernetes.io~projected/kube-api-access-2h6qd DeviceMajor:0 DeviceMinor:51 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/de6e63f3-65e8-4d4d-bad4-015fed1391a3/volumes/kubernetes.io~projected/kube-api-access-qfw88 DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/749ca676ba095a8ba9698d1d2464f22a9935fbbbd185907bd44c09179abfde60/userdata/shm DeviceMajor:0 DeviceMinor:57 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35/volumes/kubernetes.io~projected/kube-api-access-8qcz5 DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1a6cf89c-e10a-4a46-a62f-489113be6ac4/volumes/kubernetes.io~projected/kube-api-access-gfg9f DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/993fd395-dba5-4c79-99d3-72792b8c273c/volumes/kubernetes.io~projected/kube-api-access-zwqpz DeviceMajor:0 DeviceMinor:54 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/83121842cddf40c16281d4b35d0a25f62152bd2eedb22d543c870beb7d556ecc/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b460227d1567feb0c8308159d3d1155636bc75461a78280c01dc7c836da8e5af/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/560532f2-4a95-4180-a772-b269efd3d73d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2367e34dfbf292d0fff1e7b7cec1f944113d6acd7f49a860a725b7404cc10514/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c48c473c-b40e-452f-9e50-5008cddef4b4/volumes/kubernetes.io~projected/kube-api-access-rw8cx DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/560532f2-4a95-4180-a772-b269efd3d73d/volumes/kubernetes.io~projected/kube-api-access-h9tcp DeviceMajor:0 DeviceMinor:53 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/330ec43b-421e-42ee-b61e-e26743f9fbb7/volumes/kubernetes.io~projected/kube-api-access-sm9r6 DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72cc0ed4af0125f8ef16985f46ed0268c553af7fbc8805414178bf079a98ca2a/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5f8b57af621b627ef137624f5b06149995f706a4eaab762b9593d1dab152f0c/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1fe15ece-2b2c-4a7d-956f-950a1829b2ac/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/91f27c45-482c-4a05-998e-18b57ebfa499/volumes/kubernetes.io~projected/kube-api-access-rblj9 DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/21754645-32d8-4b1b-a7f5-00594f5854f3/volumes/kubernetes.io~projected/kube-api-access-6n24w DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d3524d6-1b11-41e9-9862-802d687ea87b/volumes/kubernetes.io~projected/kube-api-access-q9pf2 DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3346ce41-dd23-4e63-8944-43f2d7d4aaec/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70578d806cf234fed0c27479459e0a55693f200e9bafce2e118537e5500d9156/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df08eacd54bfadb4d947b1cc2f064e42c083f6a0048b4612fa5539ca9985b4f6/userdata/shm DeviceMajor:0 DeviceMinor:66 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/64dd929e-8d9d-47f3-bcb9-a0371d0fc279/volumes/kubernetes.io~projected/kube-api-access-xpnf2 DeviceMajor:0 DeviceMinor:52 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bbc352c4-a2e0-496b-8a9e-56b10e016a74/volumes/kubernetes.io~projected/kube-api-access-45lrv DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/560532f2-4a95-4180-a772-b269efd3d73d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a311b2965f9d8ee83128a34f312fd8708da25842aef8c59e4e7cd24ea93eed99/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a258ac5e07beeb38e563de3d745b46806e9edd50d635777975622c06721a478/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/045eb8964330e018a313575e35948e73f5c1886605fbc9eff322cbe746276a31/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fe15ece-2b2c-4a7d-956f-950a1829b2ac/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/51aeac6f-2315-42d4-b63e-47efd057cf91/volumes/kubernetes.io~projected/kube-api-access-dn99g DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/104f041ebdf17becbae4fd9723914460a4894bee6fe87894a6733eaca1791f07/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53071f197595f817a55460bb672281104ec03035e16832b108638a3111746883/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5d15af0f24c23067306c0897776d845bd443e7c7ac469369e02d09029476d44/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2367e34dfbf292d MacAddress:de:ac:5a:93:2c:08 Speed:10000 Mtu:8901} {Name:72cc0ed4af0125f MacAddress:96:16:4e:2a:c6:78 Speed:10000 Mtu:8901} {Name:83121842cddf40c MacAddress:72:59:0a:4c:6e:76 Speed:10000 Mtu:8901} {Name:a5f8b57af621b62 MacAddress:c6:c9:fa:47:d7:ee Speed:10000 Mtu:8901} {Name:b460227d1567feb MacAddress:c6:f2:fc:e8:79:de Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f8:3e:94:99 Speed:0 Mtu:9001} {Name:br-int MacAddress:f6:92:7a:21:87:67 Speed:0 Mtu:8901} {Name:e0840b7fc38d866 MacAddress:f2:1f:47:63:5a:8b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f8:3e:94:99 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:fa:73:02:44:b6:b4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7e:d4:0e:89:d6:74 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.047893 4519 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.047942 4519 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.048206 4519 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.048225 4519 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-188.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.048360 4519 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.048368 4519 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.048382 4519 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.048401 4519 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.048503 4519 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.048679 4519 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.048805 4519 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.048841 4519 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.048858 4519 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.048869 4519 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.048878 4519 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.048886 4519 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.049665 4519 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:33:24.050237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.049680 4519 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:33:24.050968 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.050310 4519 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:33:24.052255 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052241 4519 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:33:24.052290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052255 4519 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:33:24.052366 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052356 4519 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 00:33:24.052566 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052555 4519 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:33:24.052604 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052584 4519 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:33:24.052699 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052688 4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:33:24.052734 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052704 4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:33:24.052734 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052710 4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:33:24.052734 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052716 4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:33:24.052734 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052721 4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:33:24.052734 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052727 4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:33:24.052734 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052732 4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:33:24.052911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052738 4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:33:24.052911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052746 4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:33:24.052911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052752 4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:33:24.052911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052766 4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:33:24.052911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052781 4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:33:24.052911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052808 4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:33:24.052911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.052814 4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:33:24.056301 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.056282 4519 server.go:1263] "Started kubelet" Oct 31 00:33:24.056408 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.056374 4519 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:33:24.059762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.056426 4519 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:33:24.059762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.056509 4519 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:33:24.059762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.056876 4519 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:33:24.059762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.057697 4519 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:33:24.059762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.059267 4519 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:33:24.059762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.059286 4519 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:33:24.059762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.059334 4519 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:33:24.059762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.059349 4519 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:33:24.059762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.059443 4519 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:33:24.056908 ip-10-0-143-188 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:33:24.060510 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.059995 4519 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:27:45 +0000 UTC" deadline="2028-06-30 20:57:24.955322589 +0000 UTC" Oct 31 00:33:24.060510 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.060027 4519 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14612h24m0.895297863s" Oct 31 00:33:24.061703 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:24.059467 4519 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-188.us-east-2.compute.internal\" not found" Oct 31 00:33:24.063796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.063167 4519 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:33:24.063796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.063186 4519 factory.go:55] Registering systemd factory Oct 31 00:33:24.063796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.063198 4519 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:33:24.065058 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.065018 4519 factory.go:153] Registering CRI-O factory Oct 31 00:33:24.065058 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.065042 4519 factory.go:223] Registration of the crio container factory successfully Oct 31 00:33:24.065058 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.065061 4519 factory.go:103] Registering Raw factory Oct 31 00:33:24.065233 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.065076 4519 manager.go:1199] Started watching for new ooms in manager Oct 31 00:33:24.065419 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.065392 4519 manager.go:319] Starting recovery of all containers Oct 31 00:33:24.068052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.067997 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fa1579c-25f7-49b4-a9fa-e73dcaeeee35" volumeName="kubernetes.io/projected/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-kube-api-access-8qcz5" seLinuxMountContext="" Oct 31 00:33:24.068157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068056 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6e63f3-65e8-4d4d-bad4-015fed1391a3" volumeName="kubernetes.io/configmap/de6e63f3-65e8-4d4d-bad4-015fed1391a3-cni-binary-copy" seLinuxMountContext="" Oct 31 00:33:24.068157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068070 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd93b08-b857-4571-a1b5-5eaca4456f8b" volumeName="kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret" seLinuxMountContext="" Oct 31 00:33:24.068157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068083 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fe15ece-2b2c-4a7d-956f-950a1829b2ac" volumeName="kubernetes.io/secret/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 00:33:24.068157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068096 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="330ec43b-421e-42ee-b61e-e26743f9fbb7" volumeName="kubernetes.io/configmap/330ec43b-421e-42ee-b61e-e26743f9fbb7-config-volume" seLinuxMountContext="" Oct 31 00:33:24.068157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068107 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="330ec43b-421e-42ee-b61e-e26743f9fbb7" volumeName="kubernetes.io/empty-dir/330ec43b-421e-42ee-b61e-e26743f9fbb7-tmp-dir" seLinuxMountContext="" Oct 31 00:33:24.068157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068118 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3346ce41-dd23-4e63-8944-43f2d7d4aaec" volumeName="kubernetes.io/configmap/3346ce41-dd23-4e63-8944-43f2d7d4aaec-konnectivity-ca" seLinuxMountContext="" Oct 31 00:33:24.068157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068129 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51aeac6f-2315-42d4-b63e-47efd057cf91" volumeName="kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g" seLinuxMountContext="" Oct 31 00:33:24.068157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068143 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91f27c45-482c-4a05-998e-18b57ebfa499" volumeName="kubernetes.io/projected/91f27c45-482c-4a05-998e-18b57ebfa499-kube-api-access-rblj9" seLinuxMountContext="" Oct 31 00:33:24.068157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068157 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="993fd395-dba5-4c79-99d3-72792b8c273c" volumeName="kubernetes.io/empty-dir/993fd395-dba5-4c79-99d3-72792b8c273c-tmp-dir" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068168 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f037fa6-1d56-42b5-922e-007f1af8552c" volumeName="kubernetes.io/projected/9f037fa6-1d56-42b5-922e-007f1af8552c-kube-api-access-ssv6w" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068182 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d3524d6-1b11-41e9-9862-802d687ea87b" volumeName="kubernetes.io/projected/0d3524d6-1b11-41e9-9862-802d687ea87b-kube-api-access-q9pf2" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068201 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fe15ece-2b2c-4a7d-956f-950a1829b2ac" volumeName="kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-env-overrides" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068212 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21754645-32d8-4b1b-a7f5-00594f5854f3" volumeName="kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-cni-binary-copy" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068226 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f037fa6-1d56-42b5-922e-007f1af8552c" volumeName="kubernetes.io/secret/9f037fa6-1d56-42b5-922e-007f1af8552c-cert" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068239 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fa1579c-25f7-49b4-a9fa-e73dcaeeee35" volumeName="kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-service-ca-bundle" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068250 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbc352c4-a2e0-496b-8a9e-56b10e016a74" volumeName="kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068263 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6e63f3-65e8-4d4d-bad4-015fed1391a3" volumeName="kubernetes.io/projected/de6e63f3-65e8-4d4d-bad4-015fed1391a3-kube-api-access-qfw88" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068274 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fe15ece-2b2c-4a7d-956f-950a1829b2ac" volumeName="kubernetes.io/projected/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-kube-api-access-8gvtc" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068287 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64dd929e-8d9d-47f3-bcb9-a0371d0fc279" volumeName="kubernetes.io/projected/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-kube-api-access-xpnf2" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068298 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="560532f2-4a95-4180-a772-b269efd3d73d" volumeName="kubernetes.io/empty-dir/560532f2-4a95-4180-a772-b269efd3d73d-tmp" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068311 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64dd929e-8d9d-47f3-bcb9-a0371d0fc279" volumeName="kubernetes.io/configmap/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-iptables-alerter-script" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068335 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91f27c45-482c-4a05-998e-18b57ebfa499" volumeName="kubernetes.io/configmap/91f27c45-482c-4a05-998e-18b57ebfa499-serviceca" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068347 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48c473c-b40e-452f-9e50-5008cddef4b4" volumeName="kubernetes.io/projected/c48c473c-b40e-452f-9e50-5008cddef4b4-kube-api-access-rw8cx" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068360 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ec53026-ab4d-48f6-93cb-b4ce6c95ec03" volumeName="kubernetes.io/projected/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-kube-api-access-2h6qd" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068373 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21754645-32d8-4b1b-a7f5-00594f5854f3" volumeName="kubernetes.io/projected/21754645-32d8-4b1b-a7f5-00594f5854f3-kube-api-access-6n24w" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068385 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a6cf89c-e10a-4a46-a62f-489113be6ac4" volumeName="kubernetes.io/projected/1a6cf89c-e10a-4a46-a62f-489113be6ac4-kube-api-access-gfg9f" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068403 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="560532f2-4a95-4180-a772-b269efd3d73d" volumeName="kubernetes.io/projected/560532f2-4a95-4180-a772-b269efd3d73d-kube-api-access-h9tcp" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068420 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="993fd395-dba5-4c79-99d3-72792b8c273c" volumeName="kubernetes.io/projected/993fd395-dba5-4c79-99d3-72792b8c273c-kube-api-access-zwqpz" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068432 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fa1579c-25f7-49b4-a9fa-e73dcaeeee35" volumeName="kubernetes.io/empty-dir/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-archives-path" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068445 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48c473c-b40e-452f-9e50-5008cddef4b4" volumeName="kubernetes.io/secret/c48c473c-b40e-452f-9e50-5008cddef4b4-signing-key" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068461 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21754645-32d8-4b1b-a7f5-00594f5854f3" volumeName="kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068478 4519 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068497 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21754645-32d8-4b1b-a7f5-00594f5854f3" volumeName="kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068517 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3346ce41-dd23-4e63-8944-43f2d7d4aaec" volumeName="kubernetes.io/secret/3346ce41-dd23-4e63-8944-43f2d7d4aaec-agent-certs" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068532 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fa1579c-25f7-49b4-a9fa-e73dcaeeee35" volumeName="kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068546 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbc352c4-a2e0-496b-8a9e-56b10e016a74" volumeName="kubernetes.io/empty-dir/bbc352c4-a2e0-496b-8a9e-56b10e016a74-data-volume" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068558 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbc352c4-a2e0-496b-8a9e-56b10e016a74" volumeName="kubernetes.io/projected/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-api-access-45lrv" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068569 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48c473c-b40e-452f-9e50-5008cddef4b4" volumeName="kubernetes.io/configmap/c48c473c-b40e-452f-9e50-5008cddef4b4-signing-cabundle" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068582 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6e63f3-65e8-4d4d-bad4-015fed1391a3" volumeName="kubernetes.io/configmap/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-daemon-config" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068598 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fe15ece-2b2c-4a7d-956f-950a1829b2ac" volumeName="kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovnkube-config" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068611 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fe15ece-2b2c-4a7d-956f-950a1829b2ac" volumeName="kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovnkube-script-lib" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068622 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="330ec43b-421e-42ee-b61e-e26743f9fbb7" volumeName="kubernetes.io/projected/330ec43b-421e-42ee-b61e-e26743f9fbb7-kube-api-access-sm9r6" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068657 4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="560532f2-4a95-4180-a772-b269efd3d73d" volumeName="kubernetes.io/empty-dir/560532f2-4a95-4180-a772-b269efd3d73d-etc-tuned" seLinuxMountContext="" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068671 4519 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:33:24.068731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.068680 4519 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:33:24.070346 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.069133 4519 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:33:24.070346 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.070156 4519 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:33:24.076422 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.076402 4519 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 00:33:24.092857 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.092810 4519 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:33:24.094500 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.094477 4519 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:33:24.094714 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.094694 4519 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:33:24.094819 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.094731 4519 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:33:24.094819 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:24.094775 4519 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 00:33:24.101990 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.101859 4519 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:33:24.125618 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.125582 4519 generic.go:356] "Generic (PLEG): container finished" podID="72fb96218f1d813dd9872d369c087ed5" containerID="5bb9e4deae71ef44feb1d83059ea4d693559fe72c95d79081dba66e319ce1489" exitCode=0 Oct 31 00:33:24.140879 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.140807 4519 generic.go:356] "Generic (PLEG): container finished" podID="21754645-32d8-4b1b-a7f5-00594f5854f3" containerID="22dcc44dadb68207d3ed96663e3a7988bcc86794af6eb527e491915e9965f1b9" exitCode=0 Oct 31 00:33:24.140879 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.140833 4519 generic.go:356] "Generic (PLEG): container finished" podID="21754645-32d8-4b1b-a7f5-00594f5854f3" containerID="81feb16766bf180f56656b1256f6e2fac4afe6605d7907972e31a554f901d185" exitCode=0 Oct 31 00:33:24.140879 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.140840 4519 generic.go:356] "Generic (PLEG): container finished" podID="21754645-32d8-4b1b-a7f5-00594f5854f3" containerID="d9610eb58a1faf07f1cf7d046b7cbdea77cb76b00f98a75813c041122fcac570" exitCode=0 Oct 31 00:33:24.140879 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.140846 4519 generic.go:356] "Generic (PLEG): container finished" podID="21754645-32d8-4b1b-a7f5-00594f5854f3" containerID="595d3eaf77f1a61c9461bceaeafa2899e0387d9a72766ccddbe60260f98f253d" exitCode=0 Oct 31 00:33:24.140879 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.140851 4519 generic.go:356] "Generic (PLEG): container finished" podID="21754645-32d8-4b1b-a7f5-00594f5854f3" containerID="fcb6de33bc4e33ea05105d6197f8213edd3e409088440d0694311cdce30afe62" exitCode=0 Oct 31 00:33:24.140879 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.140857 4519 generic.go:356] "Generic (PLEG): container finished" podID="21754645-32d8-4b1b-a7f5-00594f5854f3" containerID="5e74368e51b6d8ab9a0142f25a79bf0153f56b02e388052e910cc0d51480bfac" exitCode=0 Oct 31 00:33:24.141165 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.140918 4519 manager.go:324] Recovery completed Oct 31 00:33:24.155819 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.155795 4519 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:33:24.155819 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.155814 4519 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:33:24.155961 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.155836 4519 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:33:24.156010 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.155992 4519 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 00:33:24.156476 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.156012 4519 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 00:33:24.156509 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.156479 4519 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 00:33:24.156509 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.156486 4519 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 00:33:24.156509 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.156492 4519 policy_none.go:50] "Start" Oct 31 00:33:24.156509 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.156503 4519 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:33:24.156613 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.156518 4519 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:33:24.156711 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.156700 4519 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 00:33:24.156743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.156711 4519 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 00:33:24.156743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.156717 4519 policy_none.go:44] "Start" Oct 31 00:33:24.160775 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.160746 4519 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:33:24.164222 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.164207 4519 manager.go:341] "Starting Device Plugin manager" Oct 31 00:33:24.164324 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:24.164239 4519 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:33:24.164324 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.164251 4519 server.go:87] "Starting device plugin registration server" Oct 31 00:33:24.164571 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.164556 4519 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:33:24.164654 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.164572 4519 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:33:24.164718 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.164670 4519 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:33:24.164800 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.164769 4519 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:33:24.164882 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.164808 4519 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:33:24.164882 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.164813 4519 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:33:24.164967 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.164877 4519 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:33:24.164796826Z","UUID":"d8fe50c4-3c83-4bf2-a771-82016a3babeb","Handler":null,"Name":"","Endpoint":""} Oct 31 00:33:24.166077 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:24.166021 4519 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:33:24.166343 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.166326 4519 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:33:24.166427 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.166359 4519 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:33:24.194933 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.194883 4519 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal"] Oct 31 00:33:24.195597 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.195570 4519 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:24.195719 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.195579 4519 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:24.195755 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.195557 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" event={"ID":"72fb96218f1d813dd9872d369c087ed5","Type":"ContainerStarted","Data":"2daad1a885385179bb48c3338384286f0ff687faa81685885b40eae74d9b7ba2"} Oct 31 00:33:24.195785 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.195765 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" event={"ID":"72fb96218f1d813dd9872d369c087ed5","Type":"ContainerDied","Data":"5bb9e4deae71ef44feb1d83059ea4d693559fe72c95d79081dba66e319ce1489"} Oct 31 00:33:24.195838 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.195790 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" event={"ID":"72fb96218f1d813dd9872d369c087ed5","Type":"ContainerStarted","Data":"749ca676ba095a8ba9698d1d2464f22a9935fbbbd185907bd44c09179abfde60"} Oct 31 00:33:24.195838 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.195816 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" event={"ID":"c7fa48eaf6754635db0f21b009865b16","Type":"ContainerStarted","Data":"0af45f65554934b364ab136e274e88d8723f45ec35c849908b023ff88ecb8904"} Oct 31 00:33:24.195838 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.195833 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" event={"ID":"c7fa48eaf6754635db0f21b009865b16","Type":"ContainerStarted","Data":"f2d4327725d3a33a287e14e7d623f858244d368269196a373031c49862b08fde"} Oct 31 00:33:24.206466 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.206446 4519 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:33:24.206522 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:24.206492 4519 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:24.207134 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.207118 4519 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:33:24.207170 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:24.207159 4519 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:24.265429 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.265394 4519 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:33:24.266732 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.266708 4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:33:24.266856 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.266744 4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:33:24.266856 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.266760 4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-188.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:33:24.266856 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.266794 4519 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:24.276625 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.276599 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72fb96218f1d813dd9872d369c087ed5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"72fb96218f1d813dd9872d369c087ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:24.276705 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.276629 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72fb96218f1d813dd9872d369c087ed5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"72fb96218f1d813dd9872d369c087ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:24.276705 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.276665 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c7fa48eaf6754635db0f21b009865b16-config\") pod \"kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"c7fa48eaf6754635db0f21b009865b16\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:24.283462 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.283437 4519 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:24.283581 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.283506 4519 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:24.377100 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.377059 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c7fa48eaf6754635db0f21b009865b16-config\") pod \"kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"c7fa48eaf6754635db0f21b009865b16\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:24.377100 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.377094 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c7fa48eaf6754635db0f21b009865b16-config\") pod \"kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"c7fa48eaf6754635db0f21b009865b16\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:24.377339 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.377129 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72fb96218f1d813dd9872d369c087ed5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"72fb96218f1d813dd9872d369c087ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:24.377339 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.377154 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72fb96218f1d813dd9872d369c087ed5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"72fb96218f1d813dd9872d369c087ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:24.377339 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.377211 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72fb96218f1d813dd9872d369c087ed5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"72fb96218f1d813dd9872d369c087ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:24.377339 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:24.377239 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72fb96218f1d813dd9872d369c087ed5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal\" (UID: \"72fb96218f1d813dd9872d369c087ed5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:25.049956 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.049921 4519 apiserver.go:52] "Watching apiserver" Oct 31 00:33:25.066542 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.066514 4519 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:33:25.067020 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.066997 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-f9mbc","openshift-cluster-node-tuning-operator/tuned-czkrf","openshift-insights/insights-runtime-extractor-pgt89","kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-l8vbc","openshift-multus/multus-6mcgm","openshift-ovn-kubernetes/ovnkube-node-h5226","kube-system/global-pull-secret-syncer-7tgpr","kube-system/konnectivity-agent-65kvb","openshift-network-diagnostics/network-check-target-6ftrv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726","openshift-dns/dns-default-72zjc","openshift-dns/node-resolver-9fz4x","openshift-ingress-canary/ingress-canary-vdxdv","openshift-network-operator/iptables-alerter-ldwwp","openshift-image-registry/node-ca-n58v5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-hkql2","openshift-insights/periodic-gathering-65vdk-w4x2d","openshift-multus/network-metrics-daemon-2662j"] Oct 31 00:33:25.070813 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.070512 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:25.073519 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.073495 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:25.074465 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.074447 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:25.088685 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.088660 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:33:25.088807 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.088747 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jwzf5\"" Oct 31 00:33:25.088852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.088813 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kxg2f\"" Oct 31 00:33:25.089435 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.089420 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:33:25.089904 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.089885 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:33:25.089949 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.089889 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:33:25.089949 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.089920 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:33:25.090146 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.090000 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k5rkn\"" Oct 31 00:33:25.090247 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.090232 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:33:25.095869 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.095851 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 00:33:25.097573 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.097557 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cvzdq\"" Oct 31 00:33:25.097809 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.097794 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:33:25.098303 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.098286 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hkzgb\"" Oct 31 00:33:25.098399 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.098305 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:33:25.098399 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.098308 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v94lw\"" Oct 31 00:33:25.098399 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.098355 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8fmps\"" Oct 31 00:33:25.099320 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.099305 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:33:25.099885 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.099869 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qhb2k\"" Oct 31 00:33:25.099987 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.099900 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:33:25.099987 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.099922 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pqlnb\"" Oct 31 00:33:25.100054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.100031 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hwf9w\"" Oct 31 00:33:25.100054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.100049 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6p62k\"" Oct 31 00:33:25.100119 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.100084 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lfknl\"" Oct 31 00:33:25.100944 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.100932 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-x5t4j\"" Oct 31 00:33:25.101024 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.101011 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 00:33:25.101024 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.101019 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jbsnc\"" Oct 31 00:33:25.101082 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.101040 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ppkcp\"" Oct 31 00:33:25.101721 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.101703 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:33:25.101809 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.101753 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:33:25.101809 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.101799 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kh74f\"" Oct 31 00:33:25.101882 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.101869 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b29sj\"" Oct 31 00:33:25.101932 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.101922 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-87knn\"" Oct 31 00:33:25.122290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.122256 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:33:25.122526 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.122504 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:33:25.122663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.122536 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 00:33:25.122663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.122553 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:33:25.122663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.122582 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:33:25.122663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.122594 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:33:25.122663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.122594 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 00:33:25.122835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.122816 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:33:25.122892 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.122841 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:33:25.122939 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.122892 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:33:25.123240 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.123225 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:33:25.123306 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.123296 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:33:25.123355 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.123340 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:33:25.123398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.123363 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 00:33:25.123398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.123384 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:33:25.124062 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.124050 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:33:25.124094 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.124074 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:33:25.124270 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.124255 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 00:33:25.124325 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.124293 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:33:25.124325 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.124304 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 00:33:25.124393 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.124349 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:33:25.124393 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.124349 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:33:25.124955 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.124929 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:33:25.125034 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.124969 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:33:25.132062 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.132038 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:33:25.132304 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.132282 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:33:25.132395 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.132328 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:33:25.132395 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.132355 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:33:25.132395 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.132355 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:33:25.133960 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.133944 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:33:25.134090 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.134071 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:33:25.134144 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.134109 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:33:25.134184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.134136 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 00:33:25.134303 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.134284 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:33:25.143539 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.143523 4519 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:25.143657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.143589 4519 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:25.158747 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.158727 4519 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:33:25.158806 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:25.158773 4519 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:25.158836 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.158731 4519 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:33:25.158889 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:25.158867 4519 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-188.us-east-2.compute.internal" Oct 31 00:33:25.161005 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.160990 4519 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:33:25.180030 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180001 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-netns\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.180151 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180068 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c48c473c-b40e-452f-9e50-5008cddef4b4-signing-cabundle\") pod \"service-ca-5fffc97bdf-f9mbc\" (UID: \"c48c473c-b40e-452f-9e50-5008cddef4b4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" Oct 31 00:33:25.180151 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180097 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rblj9\" (UniqueName: \"kubernetes.io/projected/91f27c45-482c-4a05-998e-18b57ebfa499-kube-api-access-rblj9\") pod \"node-ca-n58v5\" (UID: \"91f27c45-482c-4a05-998e-18b57ebfa499\") " pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:33:25.180151 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180121 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-system-cni-dir\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:33:25.180151 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180141 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-run-netns\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.180420 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180158 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-cni-netd\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.180420 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180175 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-host-slash\") pod \"iptables-alerter-ldwwp\" (UID: \"64dd929e-8d9d-47f3-bcb9-a0371d0fc279\") " pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:33:25.180420 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180198 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.180420 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180287 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de6e63f3-65e8-4d4d-bad4-015fed1391a3-cni-binary-copy\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.180420 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180346 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/993fd395-dba5-4c79-99d3-72792b8c273c-tmp-dir\") pod \"node-resolver-9fz4x\" (UID: \"993fd395-dba5-4c79-99d3-72792b8c273c\") " pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:33:25.180420 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180359 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c48c473c-b40e-452f-9e50-5008cddef4b4-signing-cabundle\") pod \"service-ca-5fffc97bdf-f9mbc\" (UID: \"c48c473c-b40e-452f-9e50-5008cddef4b4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" Oct 31 00:33:25.180420 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180407 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-kubernetes\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.180804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180456 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-systemd\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.180804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180456 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/993fd395-dba5-4c79-99d3-72792b8c273c-tmp-dir\") pod \"node-resolver-9fz4x\" (UID: \"993fd395-dba5-4c79-99d3-72792b8c273c\") " pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:33:25.180804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180499 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffd93b08-b857-4571-a1b5-5eaca4456f8b-kubelet-config\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:25.180804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180554 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3346ce41-dd23-4e63-8944-43f2d7d4aaec-agent-certs\") pod \"konnectivity-agent-65kvb\" (UID: \"3346ce41-dd23-4e63-8944-43f2d7d4aaec\") " pod="kube-system/konnectivity-agent-65kvb" Oct 31 00:33:25.180804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180582 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/330ec43b-421e-42ee-b61e-e26743f9fbb7-config-volume\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:25.180804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180607 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:25.180804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180649 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-registration-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.180804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180675 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-device-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.180804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180674 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de6e63f3-65e8-4d4d-bad4-015fed1391a3-cni-binary-copy\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.180804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180717 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-host\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.180804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180768 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:33:25.181294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180821 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/330ec43b-421e-42ee-b61e-e26743f9fbb7-config-volume\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:25.181294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180826 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-sys-fs\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.181294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180859 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h6qd\" (UniqueName: \"kubernetes.io/projected/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-kube-api-access-2h6qd\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.181294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180882 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9pf2\" (UniqueName: \"kubernetes.io/projected/0d3524d6-1b11-41e9-9862-802d687ea87b-kube-api-access-q9pf2\") pod \"network-metrics-daemon-2662j\" (UID: \"0d3524d6-1b11-41e9-9862-802d687ea87b\") " pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:25.181294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180917 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3346ce41-dd23-4e63-8944-43f2d7d4aaec-agent-certs\") pod \"konnectivity-agent-65kvb\" (UID: \"3346ce41-dd23-4e63-8944-43f2d7d4aaec\") " pod="kube-system/konnectivity-agent-65kvb" Oct 31 00:33:25.181294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180916 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-socket-dir-parent\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.181294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180956 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-etc-kubernetes\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.181294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.180989 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91f27c45-482c-4a05-998e-18b57ebfa499-host\") pod \"node-ca-n58v5\" (UID: \"91f27c45-482c-4a05-998e-18b57ebfa499\") " pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:33:25.181294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181018 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qcz5\" (UniqueName: \"kubernetes.io/projected/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-kube-api-access-8qcz5\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:25.181294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181045 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-cnibin\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:33:25.181294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181075 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:33:25.181294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181192 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-env-overrides\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.181294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181237 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-daemon-config\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.181748 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181308 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:33:25.181748 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181309 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssv6w\" (UniqueName: \"kubernetes.io/projected/9f037fa6-1d56-42b5-922e-007f1af8552c-kube-api-access-ssv6w\") pod \"ingress-canary-vdxdv\" (UID: \"9f037fa6-1d56-42b5-922e-007f1af8552c\") " pod="openshift-ingress-canary/ingress-canary-vdxdv" Oct 31 00:33:25.181748 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181343 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-sys\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.181748 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181359 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-etc-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.181748 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181386 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.181748 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181411 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-env-overrides\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.181748 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181426 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-daemon-config\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.181748 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181468 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovnkube-script-lib\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.181748 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181501 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-hostroot\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.181748 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181666 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3346ce41-dd23-4e63-8944-43f2d7d4aaec-konnectivity-ca\") pod \"konnectivity-agent-65kvb\" (UID: \"3346ce41-dd23-4e63-8944-43f2d7d4aaec\") " pod="kube-system/konnectivity-agent-65kvb" Oct 31 00:33:25.181748 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181689 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:33:25.181748 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181706 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovn-node-metrics-cert\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.181748 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181731 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-cni-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181733 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovnkube-script-lib\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181761 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-cni-multus\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181806 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-multus-certs\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181834 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfg9f\" (UniqueName: \"kubernetes.io/projected/1a6cf89c-e10a-4a46-a62f-489113be6ac4-kube-api-access-gfg9f\") pod \"migrator-66df5d496b-l8vbc\" (UID: \"1a6cf89c-e10a-4a46-a62f-489113be6ac4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l8vbc" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181862 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n24w\" (UniqueName: \"kubernetes.io/projected/21754645-32d8-4b1b-a7f5-00594f5854f3-kube-api-access-6n24w\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181893 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3346ce41-dd23-4e63-8944-43f2d7d4aaec-konnectivity-ca\") pod \"konnectivity-agent-65kvb\" (UID: \"3346ce41-dd23-4e63-8944-43f2d7d4aaec\") " pod="kube-system/konnectivity-agent-65kvb" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181887 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-etc-selinux\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181922 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181930 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn99g\" (UniqueName: \"kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g\") pod \"network-check-target-6ftrv\" (UID: \"51aeac6f-2315-42d4-b63e-47efd057cf91\") " pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181937 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovn-node-metrics-cert\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.181956 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw8cx\" (UniqueName: \"kubernetes.io/projected/c48c473c-b40e-452f-9e50-5008cddef4b4-kube-api-access-rw8cx\") pod \"service-ca-5fffc97bdf-f9mbc\" (UID: \"c48c473c-b40e-452f-9e50-5008cddef4b4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182076 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffd93b08-b857-4571-a1b5-5eaca4456f8b-dbus\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182112 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-ovn\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182151 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-cni-bin\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182184 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpnf2\" (UniqueName: \"kubernetes.io/projected/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-kube-api-access-xpnf2\") pod \"iptables-alerter-ldwwp\" (UID: \"64dd929e-8d9d-47f3-bcb9-a0371d0fc279\") " pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182229 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-system-cni-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182249 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-kubelet\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182268 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bbc352c4-a2e0-496b-8a9e-56b10e016a74-data-volume\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182296 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-run\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.182314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182318 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/560532f2-4a95-4180-a772-b269efd3d73d-tmp\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182340 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bbc352c4-a2e0-496b-8a9e-56b10e016a74-data-volume\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182362 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9tcp\" (UniqueName: \"kubernetes.io/projected/560532f2-4a95-4180-a772-b269efd3d73d-kube-api-access-h9tcp\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182388 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm9r6\" (UniqueName: \"kubernetes.io/projected/330ec43b-421e-42ee-b61e-e26743f9fbb7-kube-api-access-sm9r6\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182394 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/560532f2-4a95-4180-a772-b269efd3d73d-tmp\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182404 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-kubelet\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182424 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-log-socket\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182462 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-socket-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182490 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfw88\" (UniqueName: \"kubernetes.io/projected/de6e63f3-65e8-4d4d-bad4-015fed1391a3-kube-api-access-qfw88\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182513 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysconfig\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182601 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-lib-modules\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182655 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-service-ca-bundle\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182685 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/993fd395-dba5-4c79-99d3-72792b8c273c-hosts-file\") pod \"node-resolver-9fz4x\" (UID: \"993fd395-dba5-4c79-99d3-72792b8c273c\") " pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182703 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-iptables-alerter-script\") pod \"iptables-alerter-ldwwp\" (UID: \"64dd929e-8d9d-47f3-bcb9-a0371d0fc279\") " pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182725 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f037fa6-1d56-42b5-922e-007f1af8552c-cert\") pod \"ingress-canary-vdxdv\" (UID: \"9f037fa6-1d56-42b5-922e-007f1af8552c\") " pod="openshift-ingress-canary/ingress-canary-vdxdv" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182744 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-node-log\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182765 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysctl-d\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182777 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-service-ca-bundle\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182798 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwqpz\" (UniqueName: \"kubernetes.io/projected/993fd395-dba5-4c79-99d3-72792b8c273c-kube-api-access-zwqpz\") pod \"node-resolver-9fz4x\" (UID: \"993fd395-dba5-4c79-99d3-72792b8c273c\") " pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182838 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-slash\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182874 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-systemd\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182899 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182926 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bbc352c4-a2e0-496b-8a9e-56b10e016a74-crio-socket\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182930 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f037fa6-1d56-42b5-922e-007f1af8552c-cert\") pod \"ingress-canary-vdxdv\" (UID: \"9f037fa6-1d56-42b5-922e-007f1af8552c\") " pod="openshift-ingress-canary/ingress-canary-vdxdv" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182957 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182965 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-iptables-alerter-script\") pod \"iptables-alerter-ldwwp\" (UID: \"64dd929e-8d9d-47f3-bcb9-a0371d0fc279\") " pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.182991 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/330ec43b-421e-42ee-b61e-e26743f9fbb7-tmp-dir\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183020 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-conf-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183088 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183119 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45lrv\" (UniqueName: \"kubernetes.io/projected/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-api-access-45lrv\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183130 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/330ec43b-421e-42ee-b61e-e26743f9fbb7-tmp-dir\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183166 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-modprobe-d\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.183166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183182 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-var-lib-kubelet\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183194 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffd93b08-b857-4571-a1b5-5eaca4456f8b-original-pull-secret\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183213 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/560532f2-4a95-4180-a772-b269efd3d73d-etc-tuned\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183245 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91f27c45-482c-4a05-998e-18b57ebfa499-serviceca\") pod \"node-ca-n58v5\" (UID: \"91f27c45-482c-4a05-998e-18b57ebfa499\") " pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183280 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-archives-path\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183306 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-trusted-ca-bundle\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183318 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/560532f2-4a95-4180-a772-b269efd3d73d-etc-tuned\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183354 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183382 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovnkube-config\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183387 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-archives-path\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183455 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91f27c45-482c-4a05-998e-18b57ebfa499-serviceca\") pod \"node-ca-n58v5\" (UID: \"91f27c45-482c-4a05-998e-18b57ebfa499\") " pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183490 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-systemd-units\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183524 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-cnibin\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183551 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-cni-bin\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183571 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21754645-32d8-4b1b-a7f5-00594f5854f3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183570 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-trusted-ca-bundle\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183601 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183605 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-ovnkube-config\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183649 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c48c473c-b40e-452f-9e50-5008cddef4b4-signing-key\") pod \"service-ca-5fffc97bdf-f9mbc\" (UID: \"c48c473c-b40e-452f-9e50-5008cddef4b4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183685 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysctl-conf\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183750 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-var-lib-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183774 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183773 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183802 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c48c473c-b40e-452f-9e50-5008cddef4b4-signing-key\") pod \"service-ca-5fffc97bdf-f9mbc\" (UID: \"c48c473c-b40e-452f-9e50-5008cddef4b4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183820 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvtc\" (UniqueName: \"kubernetes.io/projected/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-kube-api-access-8gvtc\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183854 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs\") pod \"network-metrics-daemon-2662j\" (UID: \"0d3524d6-1b11-41e9-9862-802d687ea87b\") " pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:25.184219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.183885 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-k8s-cni-cncf-io\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.188588 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.188566 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rblj9\" (UniqueName: \"kubernetes.io/projected/91f27c45-482c-4a05-998e-18b57ebfa499-kube-api-access-rblj9\") pod \"node-ca-n58v5\" (UID: \"91f27c45-482c-4a05-998e-18b57ebfa499\") " pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:33:25.188588 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.188586 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h6qd\" (UniqueName: \"kubernetes.io/projected/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-kube-api-access-2h6qd\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.190885 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.190863 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssv6w\" (UniqueName: \"kubernetes.io/projected/9f037fa6-1d56-42b5-922e-007f1af8552c-kube-api-access-ssv6w\") pod \"ingress-canary-vdxdv\" (UID: \"9f037fa6-1d56-42b5-922e-007f1af8552c\") " pod="openshift-ingress-canary/ingress-canary-vdxdv" Oct 31 00:33:25.191149 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.191133 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfg9f\" (UniqueName: \"kubernetes.io/projected/1a6cf89c-e10a-4a46-a62f-489113be6ac4-kube-api-access-gfg9f\") pod \"migrator-66df5d496b-l8vbc\" (UID: \"1a6cf89c-e10a-4a46-a62f-489113be6ac4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l8vbc" Oct 31 00:33:25.191330 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.191314 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qcz5\" (UniqueName: \"kubernetes.io/projected/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-kube-api-access-8qcz5\") pod \"periodic-gathering-65vdk-w4x2d\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:33:25.191534 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.191520 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9pf2\" (UniqueName: \"kubernetes.io/projected/0d3524d6-1b11-41e9-9862-802d687ea87b-kube-api-access-q9pf2\") pod \"network-metrics-daemon-2662j\" (UID: \"0d3524d6-1b11-41e9-9862-802d687ea87b\") " pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:25.193698 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.193673 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm9r6\" (UniqueName: \"kubernetes.io/projected/330ec43b-421e-42ee-b61e-e26743f9fbb7-kube-api-access-sm9r6\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:25.193831 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.193812 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfw88\" (UniqueName: \"kubernetes.io/projected/de6e63f3-65e8-4d4d-bad4-015fed1391a3-kube-api-access-qfw88\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.193982 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.193948 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9tcp\" (UniqueName: \"kubernetes.io/projected/560532f2-4a95-4180-a772-b269efd3d73d-kube-api-access-h9tcp\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.194019 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.193987 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwqpz\" (UniqueName: \"kubernetes.io/projected/993fd395-dba5-4c79-99d3-72792b8c273c-kube-api-access-zwqpz\") pod \"node-resolver-9fz4x\" (UID: \"993fd395-dba5-4c79-99d3-72792b8c273c\") " pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:33:25.194184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.194171 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn99g\" (UniqueName: \"kubernetes.io/projected/51aeac6f-2315-42d4-b63e-47efd057cf91-kube-api-access-dn99g\") pod \"network-check-target-6ftrv\" (UID: \"51aeac6f-2315-42d4-b63e-47efd057cf91\") " pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:25.195530 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.195501 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpnf2\" (UniqueName: \"kubernetes.io/projected/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-kube-api-access-xpnf2\") pod \"iptables-alerter-ldwwp\" (UID: \"64dd929e-8d9d-47f3-bcb9-a0371d0fc279\") " pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:33:25.195822 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.195806 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n24w\" (UniqueName: \"kubernetes.io/projected/21754645-32d8-4b1b-a7f5-00594f5854f3-kube-api-access-6n24w\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:33:25.195903 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.195884 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw8cx\" (UniqueName: \"kubernetes.io/projected/c48c473c-b40e-452f-9e50-5008cddef4b4-kube-api-access-rw8cx\") pod \"service-ca-5fffc97bdf-f9mbc\" (UID: \"c48c473c-b40e-452f-9e50-5008cddef4b4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-f9mbc" Oct 31 00:33:25.195995 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.195981 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45lrv\" (UniqueName: \"kubernetes.io/projected/bbc352c4-a2e0-496b-8a9e-56b10e016a74-kube-api-access-45lrv\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:25.196887 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.196872 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvtc\" (UniqueName: \"kubernetes.io/projected/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-kube-api-access-8gvtc\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.284669 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284615 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-systemd-units\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.284852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284676 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-cnibin\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.284852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284695 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-cni-bin\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.284852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284711 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysctl-conf\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.284852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284724 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-systemd-units\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.284852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284725 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-var-lib-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.284852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284761 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-var-lib-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.284852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284781 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.284852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284801 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs\") pod \"network-metrics-daemon-2662j\" (UID: \"0d3524d6-1b11-41e9-9862-802d687ea87b\") " pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:25.284852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284801 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-cni-bin\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.284852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284817 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-k8s-cni-cncf-io\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.284852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284826 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.284852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284808 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-cnibin\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.284852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284834 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-netns\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284870 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-k8s-cni-cncf-io\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284873 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-netns\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284896 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-system-cni-dir\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:25.284909 4519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284911 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysctl-conf\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284927 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-run-netns\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284953 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-cni-netd\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284953 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-run-netns\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:25.284973 4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs podName:0d3524d6-1b11-41e9-9862-802d687ea87b nodeName:}" failed. No retries permitted until 2026-10-31 00:33:25.784956831 +0000 UTC m=+1.814137226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs") pod "network-metrics-daemon-2662j" (UID: "0d3524d6-1b11-41e9-9862-802d687ea87b") : secret "metrics-daemon-secret" not found Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.284965 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-system-cni-dir\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285001 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-host-slash\") pod \"iptables-alerter-ldwwp\" (UID: \"64dd929e-8d9d-47f3-bcb9-a0371d0fc279\") " pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285016 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285035 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-kubernetes\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285049 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-systemd\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285050 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64dd929e-8d9d-47f3-bcb9-a0371d0fc279-host-slash\") pod \"iptables-alerter-ldwwp\" (UID: \"64dd929e-8d9d-47f3-bcb9-a0371d0fc279\") " pod="openshift-network-operator/iptables-alerter-ldwwp" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285064 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffd93b08-b857-4571-a1b5-5eaca4456f8b-kubelet-config\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285054 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285013 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-cni-netd\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285088 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285093 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-kubernetes\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285111 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-registration-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285133 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-systemd\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285135 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-device-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285148 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffd93b08-b857-4571-a1b5-5eaca4456f8b-kubelet-config\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285173 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-host\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285187 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-registration-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285202 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285228 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-sys-fs\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285253 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-socket-dir-parent\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285264 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-host\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285278 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-etc-kubernetes\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285303 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-etc-kubernetes\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285301 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-device-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285305 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-sys-fs\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285276 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285304 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91f27c45-482c-4a05-998e-18b57ebfa499-host\") pod \"node-ca-n58v5\" (UID: \"91f27c45-482c-4a05-998e-18b57ebfa499\") " pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285335 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91f27c45-482c-4a05-998e-18b57ebfa499-host\") pod \"node-ca-n58v5\" (UID: \"91f27c45-482c-4a05-998e-18b57ebfa499\") " pod="openshift-image-registry/node-ca-n58v5" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285348 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-cnibin\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285370 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-sys\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285317 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-socket-dir-parent\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285392 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-etc-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.285398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285420 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285425 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21754645-32d8-4b1b-a7f5-00594f5854f3-cnibin\") pod \"multus-additional-cni-plugins-hkql2\" (UID: \"21754645-32d8-4b1b-a7f5-00594f5854f3\") " pod="openshift-multus/multus-additional-cni-plugins-hkql2" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285424 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-sys\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285447 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-hostroot\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285454 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-etc-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285481 4519 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285495 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-cni-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285514 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-cni-multus\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285488 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-hostroot\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285539 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-multus-certs\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285560 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-cni-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285561 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-etc-selinux\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285582 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-cni-multus\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285587 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-run-multus-certs\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285598 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffd93b08-b857-4571-a1b5-5eaca4456f8b-dbus\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285601 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-etc-selinux\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285620 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-ovn\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285483 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285658 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-cni-bin\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285680 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-ovn\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285686 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-system-cni-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285692 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffd93b08-b857-4571-a1b5-5eaca4456f8b-dbus\") pod \"global-pull-secret-syncer-7tgpr\" (UID: \"ffd93b08-b857-4571-a1b5-5eaca4456f8b\") " pod="kube-system/global-pull-secret-syncer-7tgpr" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285705 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-cni-bin\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285710 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-kubelet\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285735 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-run\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285740 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-system-cni-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285741 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-host-var-lib-kubelet\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285762 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-kubelet\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285783 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-log-socket\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285801 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-socket-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285822 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysconfig\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285835 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-log-socket\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285842 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-lib-modules\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285847 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-run\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285873 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-kubelet\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285900 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/993fd395-dba5-4c79-99d3-72792b8c273c-hosts-file\") pod \"node-resolver-9fz4x\" (UID: \"993fd395-dba5-4c79-99d3-72792b8c273c\") " pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285899 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0ec53026-ab4d-48f6-93cb-b4ce6c95ec03-socket-dir\") pod \"aws-ebs-csi-driver-node-p6726\" (UID: \"0ec53026-ab4d-48f6-93cb-b4ce6c95ec03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p6726" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285930 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-node-log\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285951 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysconfig\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285962 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-lib-modules\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.285985 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/993fd395-dba5-4c79-99d3-72792b8c273c-hosts-file\") pod \"node-resolver-9fz4x\" (UID: \"993fd395-dba5-4c79-99d3-72792b8c273c\") " pod="openshift-dns/node-resolver-9fz4x" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286001 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysctl-d\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286011 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-node-log\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286055 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-slash\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286072 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-systemd\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286086 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286101 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bbc352c4-a2e0-496b-8a9e-56b10e016a74-crio-socket\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286110 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-sysctl-d\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286117 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-conf-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286134 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286139 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-host-slash\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286152 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bbc352c4-a2e0-496b-8a9e-56b10e016a74-crio-socket\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286163 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-systemd\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286172 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-modprobe-d\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286179 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6e63f3-65e8-4d4d-bad4-015fed1391a3-multus-conf-dir\") pod \"multus-6mcgm\" (UID: \"de6e63f3-65e8-4d4d-bad4-015fed1391a3\") " pod="openshift-multus/multus-6mcgm" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286200 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-var-lib-kubelet\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286175 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1fe15ece-2b2c-4a7d-956f-950a1829b2ac-run-openvswitch\") pod \"ovnkube-node-h5226\" (UID: \"1fe15ece-2b2c-4a7d-956f-950a1829b2ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:25.286223 4519 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286272 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-var-lib-kubelet\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:25.286297 4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls podName:bbc352c4-a2e0-496b-8a9e-56b10e016a74 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:25.786279479 +0000 UTC m=+1.815459849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgt89" (UID: "bbc352c4-a2e0-496b-8a9e-56b10e016a74") : secret "insights-runtime-extractor-tls" not found Oct 31 00:33:25.286835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.286325 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/560532f2-4a95-4180-a772-b269efd3d73d-etc-modprobe-d\") pod \"tuned-czkrf\" (UID: \"560532f2-4a95-4180-a772-b269efd3d73d\") " pod="openshift-cluster-node-tuning-operator/tuned-czkrf" Oct 31 00:33:25.288405 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.287801 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/330ec43b-421e-42ee-b61e-e26743f9fbb7-metrics-tls\") pod \"dns-default-72zjc\" (UID: \"330ec43b-421e-42ee-b61e-e26743f9fbb7\") " pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:25.373851 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.373774 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:25.671199 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.671117 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-72zjc"] Oct 31 00:33:25.674615 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:25.674583 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330ec43b_421e_42ee_b61e_e26743f9fbb7.slice/crio-8612bdd87e1c454a5cd74b85cff7aab0774b6cf527db5d017d521f0738b567d1 WatchSource:0}: Error finding container 8612bdd87e1c454a5cd74b85cff7aab0774b6cf527db5d017d521f0738b567d1: Status 404 returned error can't find the container with id 8612bdd87e1c454a5cd74b85cff7aab0774b6cf527db5d017d521f0738b567d1 Oct 31 00:33:25.676698 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.676676 4519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:33:25.738034 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.737995 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:25.791135 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.791090 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs\") pod \"network-metrics-daemon-2662j\" (UID: \"0d3524d6-1b11-41e9-9862-802d687ea87b\") " pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:25.791295 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.791171 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:25.791295 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:25.791283 4519 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:33:25.791387 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:25.791340 4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls podName:bbc352c4-a2e0-496b-8a9e-56b10e016a74 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:26.791325509 +0000 UTC m=+2.820505901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgt89" (UID: "bbc352c4-a2e0-496b-8a9e-56b10e016a74") : secret "insights-runtime-extractor-tls" not found Oct 31 00:33:25.793503 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.793477 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3524d6-1b11-41e9-9862-802d687ea87b-metrics-certs\") pod \"network-metrics-daemon-2662j\" (UID: \"0d3524d6-1b11-41e9-9862-802d687ea87b\") " pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:25.947768 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.947734 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:25.971802 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:25.971778 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2662j" Oct 31 00:33:26.149317 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:26.149275 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:26.150320 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:26.150291 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-72zjc" event={"ID":"330ec43b-421e-42ee-b61e-e26743f9fbb7","Type":"ContainerStarted","Data":"8612bdd87e1c454a5cd74b85cff7aab0774b6cf527db5d017d521f0738b567d1"} Oct 31 00:33:26.273656 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:26.273614 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2662j"] Oct 31 00:33:26.278417 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:26.278388 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3524d6_1b11_41e9_9862_802d687ea87b.slice/crio-ceeab8cd6b4287404014dcece5036a976bd3c6de582787086a021651f2177109 WatchSource:0}: Error finding container ceeab8cd6b4287404014dcece5036a976bd3c6de582787086a021651f2177109: Status 404 returned error can't find the container with id ceeab8cd6b4287404014dcece5036a976bd3c6de582787086a021651f2177109 Oct 31 00:33:26.345106 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:26.345068 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:26.560106 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:26.560033 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:26.800731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:26.800693 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:26.800913 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:26.800819 4519 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:33:26.800913 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:26.800878 4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls podName:bbc352c4-a2e0-496b-8a9e-56b10e016a74 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:28.800863122 +0000 UTC m=+4.830043495 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgt89" (UID: "bbc352c4-a2e0-496b-8a9e-56b10e016a74") : secret "insights-runtime-extractor-tls" not found Oct 31 00:33:27.155267 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:27.155224 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2662j" event={"ID":"0d3524d6-1b11-41e9-9862-802d687ea87b","Type":"ContainerStarted","Data":"ceeab8cd6b4287404014dcece5036a976bd3c6de582787086a021651f2177109"} Oct 31 00:33:27.436003 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:27.435922 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9fz4x_993fd395-dba5-4c79-99d3-72792b8c273c/dns-node-resolver/0.log" Oct 31 00:33:27.581305 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:27.581277 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:27.743602 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:27.743555 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:27.945594 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:27.945558 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:28.144157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:28.144077 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:28.161111 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:28.161073 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2662j" event={"ID":"0d3524d6-1b11-41e9-9862-802d687ea87b","Type":"ContainerStarted","Data":"cb66d341b7c6730c3f9a478c62c31890f6ea28af3d510c1773b4db93b4b9e232"} Oct 31 00:33:28.161530 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:28.161141 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2662j" event={"ID":"0d3524d6-1b11-41e9-9862-802d687ea87b","Type":"ContainerStarted","Data":"79309e2681484d6296867bb729e0422767936bef4c89378c0b96bfc27a3cdea9"} Oct 31 00:33:28.162806 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:28.162783 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-72zjc" event={"ID":"330ec43b-421e-42ee-b61e-e26743f9fbb7","Type":"ContainerStarted","Data":"e6adba435f401ad61df6ad796859e4071d7d12bc012d2f0c4883e41ae3dc1b39"} Oct 31 00:33:28.162948 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:28.162829 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-72zjc" event={"ID":"330ec43b-421e-42ee-b61e-e26743f9fbb7","Type":"ContainerStarted","Data":"b4281c23a8d02222d875ac5b4fe36c877f480fb80911241113ca61f9b6b6a8cf"} Oct 31 00:33:28.209560 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:28.209515 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:28.223915 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:28.223880 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:28.303964 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:28.303930 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:28.399358 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:28.399279 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:28.543557 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:28.543523 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:28.635185 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:28.635161 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n58v5_91f27c45-482c-4a05-998e-18b57ebfa499/node-ca/0.log" Oct 31 00:33:28.820463 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:28.820423 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:28.822864 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:28.822832 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bbc352c4-a2e0-496b-8a9e-56b10e016a74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgt89\" (UID: \"bbc352c4-a2e0-496b-8a9e-56b10e016a74\") " pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:28.975169 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:28.975127 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pgt89" Oct 31 00:33:29.237922 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:29.237898 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vdxdv_9f037fa6-1d56-42b5-922e-007f1af8552c/serve-healthcheck-canary/0.log" Oct 31 00:33:29.248820 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:29.248795 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pgt89"] Oct 31 00:33:29.251779 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:29.251751 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc352c4_a2e0_496b_8a9e_56b10e016a74.slice/crio-6f15b226405fe06d3c05170d658da2cf3c9927fc34ded4756320b7c3cce609de WatchSource:0}: Error finding container 6f15b226405fe06d3c05170d658da2cf3c9927fc34ded4756320b7c3cce609de: Status 404 returned error can't find the container with id 6f15b226405fe06d3c05170d658da2cf3c9927fc34ded4756320b7c3cce609de Oct 31 00:33:29.437028 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:29.436995 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-l8vbc_1a6cf89c-e10a-4a46-a62f-489113be6ac4/migrator/0.log" Oct 31 00:33:29.749928 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:29.749890 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:29.945245 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:29.945213 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:30.168413 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:30.168381 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:30.171693 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:30.171667 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgt89" event={"ID":"bbc352c4-a2e0-496b-8a9e-56b10e016a74","Type":"ContainerStarted","Data":"5bd40b4722f4ab4f3a66a727005de5e5f2f25ed38aca72ff8217966a1ff7cc11"} Oct 31 00:33:30.171795 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:30.171740 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgt89" event={"ID":"bbc352c4-a2e0-496b-8a9e-56b10e016a74","Type":"ContainerStarted","Data":"6f15b226405fe06d3c05170d658da2cf3c9927fc34ded4756320b7c3cce609de"} Oct 31 00:33:30.349650 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:30.349599 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:30.545461 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:30.545407 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:30.842484 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:30.842404 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:31.177898 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:31.177821 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgt89" event={"ID":"bbc352c4-a2e0-496b-8a9e-56b10e016a74","Type":"ContainerStarted","Data":"c90de88cfc01a095a3d93d71712b3287d2b5f2863757faca33c00101912ce891"} Oct 31 00:33:31.583909 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:31.583881 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:31.739109 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:31.739072 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:31.836731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:31.836572 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:31.946436 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:31.946402 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:32.142897 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:32.142811 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:32.183092 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:32.183055 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgt89" event={"ID":"bbc352c4-a2e0-496b-8a9e-56b10e016a74","Type":"ContainerStarted","Data":"5b90928e614e29db6af19b49e2a52fa57a1558ca8402b2c9cff87f6f83d71c53"} Oct 31 00:33:32.290045 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:32.290002 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:32.389230 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:32.389191 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:32.471854 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:32.471778 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:32.543265 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:32.543232 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:32.610039 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:32.610015 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:32.610420 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:32.610219 4519 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:33:32.796108 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:32.796080 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:32.873322 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:32.873287 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:33.020829 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:33.020796 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:33.115661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:33.115563 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:34.368794 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.368762 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-qkcwd"] Oct 31 00:33:34.371757 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.371740 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-qkcwd" Oct 31 00:33:34.380517 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.380493 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-td7v7\"" Oct 31 00:33:34.380624 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.380562 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 00:33:34.380624 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.380576 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 00:33:34.385492 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.385468 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-qkcwd"] Oct 31 00:33:34.459946 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.459911 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95645a0e-1b05-4c37-b42b-b025b3a31ff6-tmp\") pod \"downloads-6dd789c8b7-qkcwd\" (UID: \"95645a0e-1b05-4c37-b42b-b025b3a31ff6\") " pod="openshift-console/downloads-6dd789c8b7-qkcwd" Oct 31 00:33:34.460095 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.459950 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkcpx\" (UniqueName: \"kubernetes.io/projected/95645a0e-1b05-4c37-b42b-b025b3a31ff6-kube-api-access-nkcpx\") pod \"downloads-6dd789c8b7-qkcwd\" (UID: \"95645a0e-1b05-4c37-b42b-b025b3a31ff6\") " pod="openshift-console/downloads-6dd789c8b7-qkcwd" Oct 31 00:33:34.466383 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.466354 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bb5c46fb7-x4wbn"] Oct 31 00:33:34.469435 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.469416 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.482762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.482700 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:34.484446 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.484424 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 00:33:34.484552 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.484488 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vfl7t\"" Oct 31 00:33:34.484552 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.484503 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 00:33:34.484552 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.484533 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 00:33:34.485679 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.485658 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 00:33:34.486355 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.486337 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bb5c46fb7-x4wbn"] Oct 31 00:33:34.560290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.560254 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/753e3824-d0f1-4d84-b2b2-a951de6375f9-bound-sa-token\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.560290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.560293 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkcpx\" (UniqueName: \"kubernetes.io/projected/95645a0e-1b05-4c37-b42b-b025b3a31ff6-kube-api-access-nkcpx\") pod \"downloads-6dd789c8b7-qkcwd\" (UID: \"95645a0e-1b05-4c37-b42b-b025b3a31ff6\") " pod="openshift-console/downloads-6dd789c8b7-qkcwd" Oct 31 00:33:34.560487 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.560311 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/753e3824-d0f1-4d84-b2b2-a951de6375f9-registry-tls\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.560487 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.560387 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/753e3824-d0f1-4d84-b2b2-a951de6375f9-ca-trust-extracted\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.560554 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.560494 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/753e3824-d0f1-4d84-b2b2-a951de6375f9-registry-certificates\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.560554 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.560536 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/753e3824-d0f1-4d84-b2b2-a951de6375f9-image-registry-private-configuration\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.560610 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.560597 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95645a0e-1b05-4c37-b42b-b025b3a31ff6-tmp\") pod \"downloads-6dd789c8b7-qkcwd\" (UID: \"95645a0e-1b05-4c37-b42b-b025b3a31ff6\") " pod="openshift-console/downloads-6dd789c8b7-qkcwd" Oct 31 00:33:34.560669 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.560621 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/753e3824-d0f1-4d84-b2b2-a951de6375f9-installation-pull-secrets\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.560743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.560683 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/753e3824-d0f1-4d84-b2b2-a951de6375f9-trusted-ca\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.560743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.560701 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmjx4\" (UniqueName: \"kubernetes.io/projected/753e3824-d0f1-4d84-b2b2-a951de6375f9-kube-api-access-zmjx4\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.560952 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.560935 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95645a0e-1b05-4c37-b42b-b025b3a31ff6-tmp\") pod \"downloads-6dd789c8b7-qkcwd\" (UID: \"95645a0e-1b05-4c37-b42b-b025b3a31ff6\") " pod="openshift-console/downloads-6dd789c8b7-qkcwd" Oct 31 00:33:34.570800 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.570778 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkcpx\" (UniqueName: \"kubernetes.io/projected/95645a0e-1b05-4c37-b42b-b025b3a31ff6-kube-api-access-nkcpx\") pod \"downloads-6dd789c8b7-qkcwd\" (UID: \"95645a0e-1b05-4c37-b42b-b025b3a31ff6\") " pod="openshift-console/downloads-6dd789c8b7-qkcwd" Oct 31 00:33:34.577629 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.577598 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:34.661387 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.661298 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/753e3824-d0f1-4d84-b2b2-a951de6375f9-bound-sa-token\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.661387 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.661337 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/753e3824-d0f1-4d84-b2b2-a951de6375f9-registry-tls\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.661387 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.661359 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/753e3824-d0f1-4d84-b2b2-a951de6375f9-ca-trust-extracted\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.661387 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.661377 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/753e3824-d0f1-4d84-b2b2-a951de6375f9-registry-certificates\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.661709 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.661407 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/753e3824-d0f1-4d84-b2b2-a951de6375f9-image-registry-private-configuration\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.661709 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.661438 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/753e3824-d0f1-4d84-b2b2-a951de6375f9-installation-pull-secrets\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.661709 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.661455 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/753e3824-d0f1-4d84-b2b2-a951de6375f9-trusted-ca\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.661709 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.661475 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmjx4\" (UniqueName: \"kubernetes.io/projected/753e3824-d0f1-4d84-b2b2-a951de6375f9-kube-api-access-zmjx4\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.661900 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.661861 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/753e3824-d0f1-4d84-b2b2-a951de6375f9-ca-trust-extracted\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.662902 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.662875 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/753e3824-d0f1-4d84-b2b2-a951de6375f9-registry-certificates\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.663034 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.662906 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/753e3824-d0f1-4d84-b2b2-a951de6375f9-trusted-ca\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.663996 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.663972 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/753e3824-d0f1-4d84-b2b2-a951de6375f9-image-registry-private-configuration\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.664130 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.664110 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/753e3824-d0f1-4d84-b2b2-a951de6375f9-installation-pull-secrets\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.664208 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.664195 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/753e3824-d0f1-4d84-b2b2-a951de6375f9-registry-tls\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.668769 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.668742 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/753e3824-d0f1-4d84-b2b2-a951de6375f9-bound-sa-token\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.668967 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.668949 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmjx4\" (UniqueName: \"kubernetes.io/projected/753e3824-d0f1-4d84-b2b2-a951de6375f9-kube-api-access-zmjx4\") pod \"image-registry-6bb5c46fb7-x4wbn\" (UID: \"753e3824-d0f1-4d84-b2b2-a951de6375f9\") " pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.683751 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.683733 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-qkcwd" Oct 31 00:33:34.779804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.779765 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:34.958629 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:34.958560 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-qkcwd"] Oct 31 00:33:34.961790 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:34.961760 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95645a0e_1b05_4c37_b42b_b025b3a31ff6.slice/crio-fb046b6cdda9644a9b22cb63d8473725bf6f726a80bbbb25dd873af878f09df4 WatchSource:0}: Error finding container fb046b6cdda9644a9b22cb63d8473725bf6f726a80bbbb25dd873af878f09df4: Status 404 returned error can't find the container with id fb046b6cdda9644a9b22cb63d8473725bf6f726a80bbbb25dd873af878f09df4 Oct 31 00:33:35.051973 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:35.051938 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bb5c46fb7-x4wbn"] Oct 31 00:33:35.054517 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:35.054490 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod753e3824_d0f1_4d84_b2b2_a951de6375f9.slice/crio-f7a59be7f0ac986d7207373fe0f10cc72c386a5e25bb76accb5b2fea74793fbe WatchSource:0}: Error finding container f7a59be7f0ac986d7207373fe0f10cc72c386a5e25bb76accb5b2fea74793fbe: Status 404 returned error can't find the container with id f7a59be7f0ac986d7207373fe0f10cc72c386a5e25bb76accb5b2fea74793fbe Oct 31 00:33:35.194179 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:35.194142 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-qkcwd" event={"ID":"95645a0e-1b05-4c37-b42b-b025b3a31ff6","Type":"ContainerStarted","Data":"fb046b6cdda9644a9b22cb63d8473725bf6f726a80bbbb25dd873af878f09df4"} Oct 31 00:33:35.195873 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:35.195837 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" event={"ID":"753e3824-d0f1-4d84-b2b2-a951de6375f9","Type":"ContainerStarted","Data":"a2f7637e8e06144710f49d721078ecb3793a4cc8fbc6dc5d02db2b33edeac534"} Oct 31 00:33:35.196016 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:35.195892 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" event={"ID":"753e3824-d0f1-4d84-b2b2-a951de6375f9","Type":"ContainerStarted","Data":"f7a59be7f0ac986d7207373fe0f10cc72c386a5e25bb76accb5b2fea74793fbe"} Oct 31 00:33:35.196016 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:35.195947 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:35.223477 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:35.223375 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" podStartSLOduration=1.223359462 podStartE2EDuration="1.223359462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:35.222601193 +0000 UTC m=+11.251781608" watchObservedRunningTime="2026-10-31 00:33:35.223359462 +0000 UTC m=+11.252539852" Oct 31 00:33:35.621592 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:35.621557 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:35.696287 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:35.696257 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:36.662835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:36.662799 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:36.735292 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:36.735260 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:38.073381 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:38.073346 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:38.984394 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:38.984355 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:39.083806 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:39.083771 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:39.174790 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:39.174761 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-72zjc" Oct 31 00:33:39.195874 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:39.195838 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s68fl"] Oct 31 00:33:39.199876 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:39.199347 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s68fl" Oct 31 00:33:39.208529 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:39.207938 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6gxrr\"" Oct 31 00:33:39.208529 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:39.207975 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 31 00:33:39.209088 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:39.209031 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s68fl"] Oct 31 00:33:39.212515 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:39.212489 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-65kvb" Oct 31 00:33:39.217756 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:39.217705 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:39.295743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:39.294878 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/204a65d0-1d9e-44b9-bbd5-86142f691f51-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-s68fl\" (UID: \"204a65d0-1d9e-44b9-bbd5-86142f691f51\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s68fl" Oct 31 00:33:39.396173 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:39.396122 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/204a65d0-1d9e-44b9-bbd5-86142f691f51-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-s68fl\" (UID: \"204a65d0-1d9e-44b9-bbd5-86142f691f51\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s68fl" Oct 31 00:33:39.396346 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:39.396311 4519 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Oct 31 00:33:39.396406 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:39.396393 4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/204a65d0-1d9e-44b9-bbd5-86142f691f51-tls-certificates podName:204a65d0-1d9e-44b9-bbd5-86142f691f51 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:39.896371483 +0000 UTC m=+15.925551870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/204a65d0-1d9e-44b9-bbd5-86142f691f51-tls-certificates") pod "prometheus-operator-admission-webhook-6489d8bf89-s68fl" (UID: "204a65d0-1d9e-44b9-bbd5-86142f691f51") : secret "prometheus-operator-admission-webhook-tls" not found Oct 31 00:33:39.513757 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:39.513724 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:39.900526 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:39.900484 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/204a65d0-1d9e-44b9-bbd5-86142f691f51-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-s68fl\" (UID: \"204a65d0-1d9e-44b9-bbd5-86142f691f51\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s68fl" Oct 31 00:33:39.903302 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:39.903267 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/204a65d0-1d9e-44b9-bbd5-86142f691f51-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-s68fl\" (UID: \"204a65d0-1d9e-44b9-bbd5-86142f691f51\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s68fl" Oct 31 00:33:40.115474 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:40.115437 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s68fl" Oct 31 00:33:40.398274 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:40.398238 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s68fl"] Oct 31 00:33:40.402233 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:40.402206 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204a65d0_1d9e_44b9_bbd5_86142f691f51.slice/crio-ef4c414e82b83b82aba5ef8ac6018c916c6deb949ce46878fdd3485859e01496 WatchSource:0}: Error finding container ef4c414e82b83b82aba5ef8ac6018c916c6deb949ce46878fdd3485859e01496: Status 404 returned error can't find the container with id ef4c414e82b83b82aba5ef8ac6018c916c6deb949ce46878fdd3485859e01496 Oct 31 00:33:41.220661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:41.220611 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s68fl" event={"ID":"204a65d0-1d9e-44b9-bbd5-86142f691f51","Type":"ContainerStarted","Data":"ef4c414e82b83b82aba5ef8ac6018c916c6deb949ce46878fdd3485859e01496"} Oct 31 00:33:42.226136 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:42.226051 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s68fl" event={"ID":"204a65d0-1d9e-44b9-bbd5-86142f691f51","Type":"ContainerStarted","Data":"d9de8b8fd454e35e42b4f2bd08f2266abbc087c89d2cd55a02ad5130ba25f058"} Oct 31 00:33:42.226549 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:42.226198 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s68fl" Oct 31 00:33:42.231467 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:42.231440 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s68fl" Oct 31 00:33:42.247890 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:42.247833 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s68fl" podStartSLOduration=1.820702823 podStartE2EDuration="3.24781579s" totalImagesPullingTime="1.427112967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:42.24745134 +0000 UTC m=+18.276631731" watchObservedRunningTime="2026-10-31 00:33:42.24781579 +0000 UTC m=+18.276996181" Oct 31 00:33:43.269092 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:43.269050 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:43.364191 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:43.364114 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:43.445143 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:43.445095 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:45.034069 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:45.033971 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:46.479906 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:46.479859 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:47.543541 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:33:47.543492 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:33:48.452680 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.452621 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:48.640049 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.640005 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:48.689866 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.689834 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fmstj"] Oct 31 00:33:48.720838 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.720760 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.743312 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.743245 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:48.779386 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.779351 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:33:48.779576 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.779547 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kgtx2\"" Oct 31 00:33:48.779665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.779605 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:33:48.798268 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.798235 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:33:48.798445 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.798430 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:33:48.798511 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.798454 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:33:48.798511 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.798480 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:33:48.840627 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.840578 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:48.871845 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.871807 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee44fef7-b277-4895-8ba0-def045d88a5d-node-exporter-textfile\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.872075 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.871852 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee44fef7-b277-4895-8ba0-def045d88a5d-sys\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.872075 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.871900 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee44fef7-b277-4895-8ba0-def045d88a5d-metrics-client-ca\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.872075 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.871973 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee44fef7-b277-4895-8ba0-def045d88a5d-node-exporter-tls\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.872075 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.872002 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee44fef7-b277-4895-8ba0-def045d88a5d-node-exporter-wtmp\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.872075 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.872030 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee44fef7-b277-4895-8ba0-def045d88a5d-root\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.872355 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.872138 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee44fef7-b277-4895-8ba0-def045d88a5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.872355 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.872167 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5nl6\" (UniqueName: \"kubernetes.io/projected/ee44fef7-b277-4895-8ba0-def045d88a5d-kube-api-access-b5nl6\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.872355 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.872254 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee44fef7-b277-4895-8ba0-def045d88a5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.940924 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.940886 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:48.973111 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.973017 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee44fef7-b277-4895-8ba0-def045d88a5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.973111 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.973066 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5nl6\" (UniqueName: \"kubernetes.io/projected/ee44fef7-b277-4895-8ba0-def045d88a5d-kube-api-access-b5nl6\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.973111 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.973092 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee44fef7-b277-4895-8ba0-def045d88a5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.973383 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.973123 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee44fef7-b277-4895-8ba0-def045d88a5d-node-exporter-textfile\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.973383 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.973140 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee44fef7-b277-4895-8ba0-def045d88a5d-sys\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.973383 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.973163 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee44fef7-b277-4895-8ba0-def045d88a5d-metrics-client-ca\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.973383 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.973191 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee44fef7-b277-4895-8ba0-def045d88a5d-node-exporter-tls\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.973383 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.973215 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee44fef7-b277-4895-8ba0-def045d88a5d-node-exporter-wtmp\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.973383 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.973238 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee44fef7-b277-4895-8ba0-def045d88a5d-root\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.973383 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.973313 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee44fef7-b277-4895-8ba0-def045d88a5d-root\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.973383 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.973348 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee44fef7-b277-4895-8ba0-def045d88a5d-sys\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.973784 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.973497 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee44fef7-b277-4895-8ba0-def045d88a5d-node-exporter-wtmp\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.973784 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.973588 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee44fef7-b277-4895-8ba0-def045d88a5d-node-exporter-textfile\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.974065 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.974047 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee44fef7-b277-4895-8ba0-def045d88a5d-metrics-client-ca\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.974287 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.974271 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee44fef7-b277-4895-8ba0-def045d88a5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.975800 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.975779 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee44fef7-b277-4895-8ba0-def045d88a5d-node-exporter-tls\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.975917 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.975869 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee44fef7-b277-4895-8ba0-def045d88a5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:48.985383 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:48.985348 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5nl6\" (UniqueName: \"kubernetes.io/projected/ee44fef7-b277-4895-8ba0-def045d88a5d-kube-api-access-b5nl6\") pod \"node-exporter-fmstj\" (UID: \"ee44fef7-b277-4895-8ba0-def045d88a5d\") " pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:49.031788 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:49.031749 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:49.077586 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:49.077544 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fmstj" Oct 31 00:33:49.090385 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:49.090351 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee44fef7_b277_4895_8ba0_def045d88a5d.slice/crio-faa72348bdbb212c39e34d1bc915b23da5f9f62b03f800f61a902acf1b8efd5c WatchSource:0}: Error finding container faa72348bdbb212c39e34d1bc915b23da5f9f62b03f800f61a902acf1b8efd5c: Status 404 returned error can't find the container with id faa72348bdbb212c39e34d1bc915b23da5f9f62b03f800f61a902acf1b8efd5c Oct 31 00:33:49.107519 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:49.107476 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:49.210283 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:49.210249 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:49.254822 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:49.254780 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fmstj" event={"ID":"ee44fef7-b277-4895-8ba0-def045d88a5d","Type":"ContainerStarted","Data":"faa72348bdbb212c39e34d1bc915b23da5f9f62b03f800f61a902acf1b8efd5c"} Oct 31 00:33:49.688868 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:49.688756 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:49.771631 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:49.771586 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:49.913854 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:49.913820 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:50.301902 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:50.301863 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:50.376749 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:50.376710 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:50.830908 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:50.830871 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:50.906359 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:50.906331 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:51.263453 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:51.263419 4519 generic.go:356] "Generic (PLEG): container finished" podID="ee44fef7-b277-4895-8ba0-def045d88a5d" containerID="3772bc5f11bf2e18be09ba88a9675749bca54ebc42a089bd48b791990a32ddf4" exitCode=0 Oct 31 00:33:51.263618 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:51.263507 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fmstj" event={"ID":"ee44fef7-b277-4895-8ba0-def045d88a5d","Type":"ContainerDied","Data":"3772bc5f11bf2e18be09ba88a9675749bca54ebc42a089bd48b791990a32ddf4"} Oct 31 00:33:51.654486 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:51.654395 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:51.738877 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:51.738834 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:51.869494 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:51.869459 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:51.969179 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:51.969099 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:52.270713 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:52.270671 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fmstj" event={"ID":"ee44fef7-b277-4895-8ba0-def045d88a5d","Type":"ContainerStarted","Data":"16eceb79b432eb79e21c92e48f9b81055157e06aa93fc69ac7d96f67183d071c"} Oct 31 00:33:52.270916 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:52.270745 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fmstj" event={"ID":"ee44fef7-b277-4895-8ba0-def045d88a5d","Type":"ContainerStarted","Data":"9cd58a2b857af8c77b50642c7c8086f4c59fd1757bd23816755c288cbf519f1d"} Oct 31 00:33:52.272859 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:52.272825 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-qkcwd" event={"ID":"95645a0e-1b05-4c37-b42b-b025b3a31ff6","Type":"ContainerStarted","Data":"c932e18d86232ce6cb059ee799cb29aedd7bf67bace51fa16f36bca7f697452b"} Oct 31 00:33:52.273071 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:52.273047 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-qkcwd" Oct 31 00:33:52.289842 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:52.289808 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-qkcwd" Oct 31 00:33:52.295271 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:52.295208 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fmstj" podStartSLOduration=2.465947658 podStartE2EDuration="4.295192731s" totalImagesPullingTime="1.755901716s" totalInitContainerRuntime="73.343357ms" isStatefulPod=true podCreationTimestamp="2026-10-31 00:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:52.294239388 +0000 UTC m=+28.323419780" watchObservedRunningTime="2026-10-31 00:33:52.295192731 +0000 UTC m=+28.324373122" Oct 31 00:33:52.939115 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:52.939086 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5226" Oct 31 00:33:52.976197 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:52.976133 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-qkcwd" podStartSLOduration=2.610378786 podStartE2EDuration="18.97611292s" totalImagesPullingTime="16.365734134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:52.318871834 +0000 UTC m=+28.348052208" watchObservedRunningTime="2026-10-31 00:33:52.97611292 +0000 UTC m=+29.005293309" Oct 31 00:33:53.092767 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.092730 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-d969cdb45-lvjhp"] Oct 31 00:33:53.127081 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.127046 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d969cdb45-lvjhp"] Oct 31 00:33:53.127284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.127178 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.132020 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.131993 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 00:33:53.132168 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.132023 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 00:33:53.132168 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.132073 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 00:33:53.132168 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.132144 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-alue0d2p97la4\"" Oct 31 00:33:53.132315 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.132171 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rxns2\"" Oct 31 00:33:53.132315 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.132199 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 00:33:53.215808 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.215714 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc4tf\" (UniqueName: \"kubernetes.io/projected/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-kube-api-access-kc4tf\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.215808 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.215804 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-client-ca-bundle\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.216046 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.215887 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-secret-metrics-server-client-certs\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.216046 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.215954 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-metrics-server-audit-profiles\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.216046 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.215980 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-secret-metrics-server-tls\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.216046 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.216017 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.216207 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.216107 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-audit-log\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.317282 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.317242 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-secret-metrics-server-client-certs\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.317486 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.317300 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-metrics-server-audit-profiles\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.317486 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.317326 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-secret-metrics-server-tls\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.317486 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.317351 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.317486 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.317377 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-audit-log\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.317486 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.317430 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc4tf\" (UniqueName: \"kubernetes.io/projected/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-kube-api-access-kc4tf\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.317769 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.317512 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-client-ca-bundle\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.318033 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.317999 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-audit-log\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.318716 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.318682 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-metrics-server-audit-profiles\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.319158 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.319130 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.320513 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.320467 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-client-ca-bundle\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.320683 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.320660 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-secret-metrics-server-tls\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.321706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.321682 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-secret-metrics-server-client-certs\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.326173 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.326151 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc4tf\" (UniqueName: \"kubernetes.io/projected/76a22588-6b4d-4c35-b00a-eaae2cff3ae7-kube-api-access-kc4tf\") pod \"metrics-server-d969cdb45-lvjhp\" (UID: \"76a22588-6b4d-4c35-b00a-eaae2cff3ae7\") " pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.457452 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.457407 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:33:53.608825 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.608790 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:53.685835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.685800 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:53.783813 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.783741 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d969cdb45-lvjhp"] Oct 31 00:33:53.787374 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:53.787340 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a22588_6b4d_4c35_b00a_eaae2cff3ae7.slice/crio-a157210a618e120e901c920e2dc673e18384824e731f3328579b3b5705eb4a3a WatchSource:0}: Error finding container a157210a618e120e901c920e2dc673e18384824e731f3328579b3b5705eb4a3a: Status 404 returned error can't find the container with id a157210a618e120e901c920e2dc673e18384824e731f3328579b3b5705eb4a3a Oct 31 00:33:53.934917 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:53.934876 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:54.035431 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:54.035341 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:54.282766 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:54.282727 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" event={"ID":"76a22588-6b4d-4c35-b00a-eaae2cff3ae7","Type":"ContainerStarted","Data":"a157210a618e120e901c920e2dc673e18384824e731f3328579b3b5705eb4a3a"} Oct 31 00:33:54.642046 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:54.642006 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:55.006495 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:55.006398 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:56.204130 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:56.204053 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bb5c46fb7-x4wbn" Oct 31 00:33:56.297516 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:56.297479 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" event={"ID":"76a22588-6b4d-4c35-b00a-eaae2cff3ae7","Type":"ContainerStarted","Data":"aa3f83db48ce7a44058c572e05fc8dfce1d404ab374094da741f7d56c6415c19"} Oct 31 00:33:56.321393 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:56.321336 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" podStartSLOduration=1.220071843 podStartE2EDuration="3.32131646s" totalImagesPullingTime="2.101244617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:56.320546577 +0000 UTC m=+32.349726967" watchObservedRunningTime="2026-10-31 00:33:56.32131646 +0000 UTC m=+32.350496846" Oct 31 00:33:56.369537 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:56.369499 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:56.469852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:56.469768 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:57.044716 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.044679 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-57cb58b78-cpwpv"] Oct 31 00:33:57.076425 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.076389 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57cb58b78-cpwpv"] Oct 31 00:33:57.076596 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.076405 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.082756 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.082727 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 00:33:57.084490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.084466 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 00:33:57.084600 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.084564 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 00:33:57.084691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.084576 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 00:33:57.084691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.084617 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 00:33:57.084691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.084615 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mv6rq\"" Oct 31 00:33:57.091470 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.091451 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 00:33:57.253864 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.253828 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gcj2\" (UniqueName: \"kubernetes.io/projected/a6079d2f-a234-49c1-9011-57cbba17a01f-kube-api-access-8gcj2\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.254326 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.253901 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-service-ca\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.254326 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.254011 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6079d2f-a234-49c1-9011-57cbba17a01f-tmp\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.254326 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.254063 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6079d2f-a234-49c1-9011-57cbba17a01f-console-serving-cert\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.254326 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.254124 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-console-config\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.254326 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.254167 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6079d2f-a234-49c1-9011-57cbba17a01f-console-oauth-config\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.254326 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.254226 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-trusted-ca-bundle\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.254326 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.254252 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-oauth-serving-cert\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.354996 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.354911 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-service-ca\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.354996 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.354951 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6079d2f-a234-49c1-9011-57cbba17a01f-tmp\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.354996 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.354982 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6079d2f-a234-49c1-9011-57cbba17a01f-console-serving-cert\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.355265 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.355037 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-console-config\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.355265 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.355096 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6079d2f-a234-49c1-9011-57cbba17a01f-console-oauth-config\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.355265 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.355134 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-trusted-ca-bundle\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.355265 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.355157 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-oauth-serving-cert\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.355265 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.355195 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gcj2\" (UniqueName: \"kubernetes.io/projected/a6079d2f-a234-49c1-9011-57cbba17a01f-kube-api-access-8gcj2\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.355519 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.355378 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6079d2f-a234-49c1-9011-57cbba17a01f-tmp\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.355905 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.355878 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-service-ca\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.356016 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.355934 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-console-config\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.356219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.356170 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-oauth-serving-cert\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.356326 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.356245 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-trusted-ca-bundle\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.357824 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.357802 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6079d2f-a234-49c1-9011-57cbba17a01f-console-serving-cert\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.358146 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.358118 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6079d2f-a234-49c1-9011-57cbba17a01f-console-oauth-config\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.364489 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.364466 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gcj2\" (UniqueName: \"kubernetes.io/projected/a6079d2f-a234-49c1-9011-57cbba17a01f-kube-api-access-8gcj2\") pod \"console-57cb58b78-cpwpv\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.412978 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.412943 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:33:57.584811 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.584776 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6ftrv" Oct 31 00:33:57.728119 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:57.728038 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57cb58b78-cpwpv"] Oct 31 00:33:57.732433 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:33:57.732407 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6079d2f_a234_49c1_9011_57cbba17a01f.slice/crio-1c2cf4b7331070a06fa318a7956a0f586b56ca8454ad84604cc4f84138ec088f WatchSource:0}: Error finding container 1c2cf4b7331070a06fa318a7956a0f586b56ca8454ad84604cc4f84138ec088f: Status 404 returned error can't find the container with id 1c2cf4b7331070a06fa318a7956a0f586b56ca8454ad84604cc4f84138ec088f Oct 31 00:33:58.306437 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:58.306399 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57cb58b78-cpwpv" event={"ID":"a6079d2f-a234-49c1-9011-57cbba17a01f","Type":"ContainerStarted","Data":"1c2cf4b7331070a06fa318a7956a0f586b56ca8454ad84604cc4f84138ec088f"} Oct 31 00:33:59.436964 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:59.436923 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:59.538343 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:33:59.538307 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:00.082490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:00.082432 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:00.177064 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:00.177021 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:01.417253 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:01.417225 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:01.517807 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:01.517770 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:01.620390 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:01.620288 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:01.719077 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:01.719042 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:02.323673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:02.323623 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57cb58b78-cpwpv" event={"ID":"a6079d2f-a234-49c1-9011-57cbba17a01f","Type":"ContainerStarted","Data":"4c4aebff9a822cda9734112b7bafa7706cf2463687ce426d88f9c3ea2630a557"} Oct 31 00:34:02.353045 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:02.352990 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-57cb58b78-cpwpv" podStartSLOduration=1.743104626 podStartE2EDuration="5.352975057s" totalImagesPullingTime="3.609870431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:02.352234551 +0000 UTC m=+38.381414953" watchObservedRunningTime="2026-10-31 00:34:02.352975057 +0000 UTC m=+38.382155447" Oct 31 00:34:04.395087 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:04.395037 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:04.467738 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:04.467692 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:07.413414 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:07.413332 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:34:07.413414 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:07.413396 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:34:07.418141 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:07.418118 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:34:08.140512 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:08.140477 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:08.242463 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:08.242417 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:08.346795 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:08.346770 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:34:09.772328 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:09.772281 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:09.870250 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:09.870222 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:13.458437 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:13.458397 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:34:13.458898 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:13.458482 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:34:17.540731 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:34:17.540695 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:34:21.651091 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:21.651053 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:21.749432 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:21.749397 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:33.462690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:33.462662 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:34:33.466409 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:33.466383 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-d969cdb45-lvjhp" Oct 31 00:34:36.657074 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:36.657039 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:36.728010 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:36.727966 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:37.992961 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:37.992923 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:38.091167 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:38.091136 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:45.463383 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:45.463348 4519 generic.go:356] "Generic (PLEG): container finished" podID="9fa1579c-25f7-49b4-a9fa-e73dcaeeee35" containerID="f503f1766c023f8399506061e61349cc6e9388be692a0560c0039fe55393c728" exitCode=0 Oct 31 00:34:45.463928 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:45.463426 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-65vdk-w4x2d" event={"ID":"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35","Type":"ContainerDied","Data":"f503f1766c023f8399506061e61349cc6e9388be692a0560c0039fe55393c728"} Oct 31 00:34:46.427218 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.427179 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d68bbc5f4-4m6gx"] Oct 31 00:34:46.430066 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.430043 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.441038 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.441010 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d68bbc5f4-4m6gx"] Oct 31 00:34:46.494922 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.494888 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:46.511606 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.511567 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-oauth-serving-cert\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.511773 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.511614 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-trusted-ca-bundle\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.511814 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.511771 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-config\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.511848 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.511822 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-service-ca\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.511881 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.511858 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-tmp\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.511910 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.511882 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-oauth-config\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.512001 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.511984 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-serving-cert\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.512041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.512008 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbgm2\" (UniqueName: \"kubernetes.io/projected/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-kube-api-access-zbgm2\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.587559 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.587532 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:34:46.594716 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.594690 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:46.611596 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.611562 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-65vdk-w4x2d"] Oct 31 00:34:46.612965 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.612944 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-service-ca-bundle\") pod \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " Oct 31 00:34:46.613035 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.612982 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-archives-path\") pod \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " Oct 31 00:34:46.613035 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.613015 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-trusted-ca-bundle\") pod \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " Oct 31 00:34:46.613137 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.613038 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-kube-api-access-8qcz5\" (UniqueName: \"kubernetes.io/projected/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-kube-api-access-8qcz5\") pod \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\" (UID: \"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35\") " Oct 31 00:34:46.613137 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.613118 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-oauth-serving-cert\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.613235 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.613147 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-trusted-ca-bundle\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.613235 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.613202 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-config\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.613235 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.613230 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-service-ca\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.613386 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.613266 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-tmp\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.613386 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.613291 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-oauth-config\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.613386 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.613344 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-serving-cert\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.613386 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.613370 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbgm2\" (UniqueName: \"kubernetes.io/projected/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-kube-api-access-zbgm2\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.613559 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.613374 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-archives-path" pod "9fa1579c-25f7-49b4-a9fa-e73dcaeeee35" (UID: "9fa1579c-25f7-49b4-a9fa-e73dcaeeee35"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:34:46.613559 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.613418 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-service-ca-bundle" pod "9fa1579c-25f7-49b4-a9fa-e73dcaeeee35" (UID: "9fa1579c-25f7-49b4-a9fa-e73dcaeeee35"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:46.613559 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.613427 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-trusted-ca-bundle" pod "9fa1579c-25f7-49b4-a9fa-e73dcaeeee35" (UID: "9fa1579c-25f7-49b4-a9fa-e73dcaeeee35"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:46.613976 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.613951 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-oauth-serving-cert\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.614345 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.614321 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-tmp\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.614429 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.614321 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-trusted-ca-bundle\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.614532 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.614510 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-service-ca\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.614787 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.614769 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-config\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.615767 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.615737 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-kube-api-access-8qcz5" pod "9fa1579c-25f7-49b4-a9fa-e73dcaeeee35" (UID: "9fa1579c-25f7-49b4-a9fa-e73dcaeeee35"). InnerVolumeSpecName "kube-api-access-8qcz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:46.616045 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.616028 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-serving-cert\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.616113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.616103 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-oauth-config\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.628219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.628197 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbgm2\" (UniqueName: \"kubernetes.io/projected/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-kube-api-access-zbgm2\") pod \"console-6d68bbc5f4-4m6gx\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.680254 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.680178 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-fg9fp-xtjlp"] Oct 31 00:34:46.680432 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.680420 4519 cpu_manager.go:414] "containerMap: removing container" podUID="9fa1579c-25f7-49b4-a9fa-e73dcaeeee35" containerName="insights-gathering" Oct 31 00:34:46.680472 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.680433 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa1579c-25f7-49b4-a9fa-e73dcaeeee35" containerName="insights-gathering" Oct 31 00:34:46.680505 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.680489 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa1579c-25f7-49b4-a9fa-e73dcaeeee35" containerName="insights-gathering" Oct 31 00:34:46.684090 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.684074 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" Oct 31 00:34:46.691517 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.691313 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fg9fp-xtjlp"] Oct 31 00:34:46.714594 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.714565 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4c46\" (UniqueName: \"kubernetes.io/projected/623173be-674f-4b4c-8920-d8b1f213666b-kube-api-access-d4c46\") pod \"periodic-gathering-fg9fp-xtjlp\" (UID: \"623173be-674f-4b4c-8920-d8b1f213666b\") " pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" Oct 31 00:34:46.714775 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.714601 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/623173be-674f-4b4c-8920-d8b1f213666b-archives-path\") pod \"periodic-gathering-fg9fp-xtjlp\" (UID: \"623173be-674f-4b4c-8920-d8b1f213666b\") " pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" Oct 31 00:34:46.714775 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.714620 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/623173be-674f-4b4c-8920-d8b1f213666b-trusted-ca-bundle\") pod \"periodic-gathering-fg9fp-xtjlp\" (UID: \"623173be-674f-4b4c-8920-d8b1f213666b\") " pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" Oct 31 00:34:46.714775 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.714736 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/623173be-674f-4b4c-8920-d8b1f213666b-service-ca-bundle\") pod \"periodic-gathering-fg9fp-xtjlp\" (UID: \"623173be-674f-4b4c-8920-d8b1f213666b\") " pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" Oct 31 00:34:46.714775 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.714764 4519 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-service-ca-bundle\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:46.714775 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.714776 4519 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-archives-path\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:46.714936 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.714787 4519 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-trusted-ca-bundle\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:46.714936 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.714798 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qcz5\" (UniqueName: \"kubernetes.io/projected/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35-kube-api-access-8qcz5\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:46.740613 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.740580 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:46.815440 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.815407 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/623173be-674f-4b4c-8920-d8b1f213666b-service-ca-bundle\") pod \"periodic-gathering-fg9fp-xtjlp\" (UID: \"623173be-674f-4b4c-8920-d8b1f213666b\") " pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" Oct 31 00:34:46.815591 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.815453 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4c46\" (UniqueName: \"kubernetes.io/projected/623173be-674f-4b4c-8920-d8b1f213666b-kube-api-access-d4c46\") pod \"periodic-gathering-fg9fp-xtjlp\" (UID: \"623173be-674f-4b4c-8920-d8b1f213666b\") " pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" Oct 31 00:34:46.815591 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.815476 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/623173be-674f-4b4c-8920-d8b1f213666b-archives-path\") pod \"periodic-gathering-fg9fp-xtjlp\" (UID: \"623173be-674f-4b4c-8920-d8b1f213666b\") " pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" Oct 31 00:34:46.815591 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.815494 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/623173be-674f-4b4c-8920-d8b1f213666b-trusted-ca-bundle\") pod \"periodic-gathering-fg9fp-xtjlp\" (UID: \"623173be-674f-4b4c-8920-d8b1f213666b\") " pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" Oct 31 00:34:46.815955 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.815932 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/623173be-674f-4b4c-8920-d8b1f213666b-archives-path\") pod \"periodic-gathering-fg9fp-xtjlp\" (UID: \"623173be-674f-4b4c-8920-d8b1f213666b\") " pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" Oct 31 00:34:46.816247 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.816230 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/623173be-674f-4b4c-8920-d8b1f213666b-service-ca-bundle\") pod \"periodic-gathering-fg9fp-xtjlp\" (UID: \"623173be-674f-4b4c-8920-d8b1f213666b\") " pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" Oct 31 00:34:46.816457 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.816441 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/623173be-674f-4b4c-8920-d8b1f213666b-trusted-ca-bundle\") pod \"periodic-gathering-fg9fp-xtjlp\" (UID: \"623173be-674f-4b4c-8920-d8b1f213666b\") " pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" Oct 31 00:34:46.824993 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.824970 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4c46\" (UniqueName: \"kubernetes.io/projected/623173be-674f-4b4c-8920-d8b1f213666b-kube-api-access-d4c46\") pod \"periodic-gathering-fg9fp-xtjlp\" (UID: \"623173be-674f-4b4c-8920-d8b1f213666b\") " pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" Oct 31 00:34:46.996985 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:46.996942 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" Oct 31 00:34:47.009423 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:47.009398 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d68bbc5f4-4m6gx"] Oct 31 00:34:47.011854 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:34:47.011827 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4e3f3f_d16d_49e2_a862_f5be9e43d4ed.slice/crio-5cca55460ad9752386d5ac36c9d8b492851a8c5118da0ce3f370ff82ef013693 WatchSource:0}: Error finding container 5cca55460ad9752386d5ac36c9d8b492851a8c5118da0ce3f370ff82ef013693: Status 404 returned error can't find the container with id 5cca55460ad9752386d5ac36c9d8b492851a8c5118da0ce3f370ff82ef013693 Oct 31 00:34:47.116725 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:47.116687 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:47.215781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:47.215751 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:47.277859 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:47.277714 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fg9fp-xtjlp"] Oct 31 00:34:47.280261 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:34:47.280235 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623173be_674f_4b4c_8920_d8b1f213666b.slice/crio-0b2de6e6c23422033c2f4de7caefceac127006bf288f28337df8b76008164ef3 WatchSource:0}: Error finding container 0b2de6e6c23422033c2f4de7caefceac127006bf288f28337df8b76008164ef3: Status 404 returned error can't find the container with id 0b2de6e6c23422033c2f4de7caefceac127006bf288f28337df8b76008164ef3 Oct 31 00:34:47.471040 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:47.471007 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-65vdk-w4x2d" event={"ID":"9fa1579c-25f7-49b4-a9fa-e73dcaeeee35","Type":"ContainerDied","Data":"72cc0ed4af0125f8ef16985f46ed0268c553af7fbc8805414178bf079a98ca2a"} Oct 31 00:34:47.471040 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:47.471026 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-65vdk-w4x2d" Oct 31 00:34:47.471266 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:47.471064 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72cc0ed4af0125f8ef16985f46ed0268c553af7fbc8805414178bf079a98ca2a" Oct 31 00:34:47.472570 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:47.472540 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d68bbc5f4-4m6gx" event={"ID":"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed","Type":"ContainerStarted","Data":"0bbb4ac4986aea14bfb9d4c26fd5e353f37d800270353c05824cfac9c7f5d754"} Oct 31 00:34:47.472739 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:47.472589 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d68bbc5f4-4m6gx" event={"ID":"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed","Type":"ContainerStarted","Data":"5cca55460ad9752386d5ac36c9d8b492851a8c5118da0ce3f370ff82ef013693"} Oct 31 00:34:47.474064 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:47.474037 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" event={"ID":"623173be-674f-4b4c-8920-d8b1f213666b","Type":"ContainerStarted","Data":"c517725c8286577dba9f7bae952f16965121576a654f99fcfea5477cc4c36fd2"} Oct 31 00:34:47.474175 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:47.474083 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" event={"ID":"623173be-674f-4b4c-8920-d8b1f213666b","Type":"ContainerStarted","Data":"0b2de6e6c23422033c2f4de7caefceac127006bf288f28337df8b76008164ef3"} Oct 31 00:34:47.493610 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:47.493565 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6d68bbc5f4-4m6gx" podStartSLOduration=1.493551349 podStartE2EDuration="1.493551349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:47.493205913 +0000 UTC m=+83.522386303" watchObservedRunningTime="2026-10-31 00:34:47.493551349 +0000 UTC m=+83.522731739" Oct 31 00:34:47.506117 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:47.506079 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-65vdk-w4x2d"] Oct 31 00:34:47.510929 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:47.510901 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-65vdk-w4x2d"] Oct 31 00:34:47.526099 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:47.526046 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" podStartSLOduration=1.526031853 podStartE2EDuration="1.526031853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:47.525973697 +0000 UTC m=+83.555154094" watchObservedRunningTime="2026-10-31 00:34:47.526031853 +0000 UTC m=+83.555212242" Oct 31 00:34:47.541910 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:34:47.541828 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:34:48.100152 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:48.100117 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa1579c-25f7-49b4-a9fa-e73dcaeeee35" path="/var/lib/kubelet/pods/9fa1579c-25f7-49b4-a9fa-e73dcaeeee35/volumes" Oct 31 00:34:56.740742 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:56.740705 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:56.741255 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:56.740770 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:56.745758 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:56.745735 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:57.511835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:57.511805 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:34:57.561099 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:34:57.561061 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57cb58b78-cpwpv"] Oct 31 00:35:09.818110 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:09.818077 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:17.540535 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:35:17.540495 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:35:22.581077 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.581013 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-57cb58b78-cpwpv" podUID="a6079d2f-a234-49c1-9011-57cbba17a01f" containerName="console" containerID="cri-o://4c4aebff9a822cda9734112b7bafa7706cf2463687ce426d88f9c3ea2630a557" gracePeriod=15 Oct 31 00:35:22.820153 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.820125 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:35:22.911164 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.911073 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-console-config\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-console-config\") pod \"a6079d2f-a234-49c1-9011-57cbba17a01f\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " Oct 31 00:35:22.911164 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.911129 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6079d2f-a234-49c1-9011-57cbba17a01f-kube-api-access-8gcj2\" (UniqueName: \"kubernetes.io/projected/a6079d2f-a234-49c1-9011-57cbba17a01f-kube-api-access-8gcj2\") pod \"a6079d2f-a234-49c1-9011-57cbba17a01f\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " Oct 31 00:35:22.911164 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.911153 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6079d2f-a234-49c1-9011-57cbba17a01f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6079d2f-a234-49c1-9011-57cbba17a01f-console-oauth-config\") pod \"a6079d2f-a234-49c1-9011-57cbba17a01f\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " Oct 31 00:35:22.911631 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.911172 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6079d2f-a234-49c1-9011-57cbba17a01f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6079d2f-a234-49c1-9011-57cbba17a01f-tmp\") pod \"a6079d2f-a234-49c1-9011-57cbba17a01f\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " Oct 31 00:35:22.911631 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.911201 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-trusted-ca-bundle\") pod \"a6079d2f-a234-49c1-9011-57cbba17a01f\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " Oct 31 00:35:22.911631 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.911222 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-service-ca\") pod \"a6079d2f-a234-49c1-9011-57cbba17a01f\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " Oct 31 00:35:22.911631 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.911273 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-oauth-serving-cert\") pod \"a6079d2f-a234-49c1-9011-57cbba17a01f\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " Oct 31 00:35:22.911631 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.911301 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6079d2f-a234-49c1-9011-57cbba17a01f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6079d2f-a234-49c1-9011-57cbba17a01f-console-serving-cert\") pod \"a6079d2f-a234-49c1-9011-57cbba17a01f\" (UID: \"a6079d2f-a234-49c1-9011-57cbba17a01f\") " Oct 31 00:35:22.911631 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.911501 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6079d2f-a234-49c1-9011-57cbba17a01f-tmp" pod "a6079d2f-a234-49c1-9011-57cbba17a01f" (UID: "a6079d2f-a234-49c1-9011-57cbba17a01f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:22.911966 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.911670 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-trusted-ca-bundle" pod "a6079d2f-a234-49c1-9011-57cbba17a01f" (UID: "a6079d2f-a234-49c1-9011-57cbba17a01f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:22.911966 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.911750 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-service-ca" pod "a6079d2f-a234-49c1-9011-57cbba17a01f" (UID: "a6079d2f-a234-49c1-9011-57cbba17a01f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:22.911966 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.911762 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-oauth-serving-cert" pod "a6079d2f-a234-49c1-9011-57cbba17a01f" (UID: "a6079d2f-a234-49c1-9011-57cbba17a01f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:22.912090 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.911964 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-console-config" pod "a6079d2f-a234-49c1-9011-57cbba17a01f" (UID: "a6079d2f-a234-49c1-9011-57cbba17a01f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:22.913520 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.913495 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6079d2f-a234-49c1-9011-57cbba17a01f-console-oauth-config" pod "a6079d2f-a234-49c1-9011-57cbba17a01f" (UID: "a6079d2f-a234-49c1-9011-57cbba17a01f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:22.913604 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.913522 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6079d2f-a234-49c1-9011-57cbba17a01f-console-serving-cert" pod "a6079d2f-a234-49c1-9011-57cbba17a01f" (UID: "a6079d2f-a234-49c1-9011-57cbba17a01f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:22.913604 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:22.913495 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6079d2f-a234-49c1-9011-57cbba17a01f-kube-api-access-8gcj2" pod "a6079d2f-a234-49c1-9011-57cbba17a01f" (UID: "a6079d2f-a234-49c1-9011-57cbba17a01f"). InnerVolumeSpecName "kube-api-access-8gcj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:23.011945 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:23.011903 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gcj2\" (UniqueName: \"kubernetes.io/projected/a6079d2f-a234-49c1-9011-57cbba17a01f-kube-api-access-8gcj2\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:23.011945 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:23.011937 4519 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6079d2f-a234-49c1-9011-57cbba17a01f-console-oauth-config\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:23.011945 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:23.011951 4519 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6079d2f-a234-49c1-9011-57cbba17a01f-tmp\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:23.012195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:23.011963 4519 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-trusted-ca-bundle\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:23.012195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:23.011975 4519 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-service-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:23.012195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:23.011987 4519 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-oauth-serving-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:23.012195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:23.011998 4519 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6079d2f-a234-49c1-9011-57cbba17a01f-console-serving-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:23.012195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:23.012009 4519 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6079d2f-a234-49c1-9011-57cbba17a01f-console-config\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:23.596865 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:23.596831 4519 generic.go:356] "Generic (PLEG): container finished" podID="a6079d2f-a234-49c1-9011-57cbba17a01f" containerID="4c4aebff9a822cda9734112b7bafa7706cf2463687ce426d88f9c3ea2630a557" exitCode=0 Oct 31 00:35:23.597347 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:23.596873 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57cb58b78-cpwpv" event={"ID":"a6079d2f-a234-49c1-9011-57cbba17a01f","Type":"ContainerDied","Data":"4c4aebff9a822cda9734112b7bafa7706cf2463687ce426d88f9c3ea2630a557"} Oct 31 00:35:23.597347 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:23.596881 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57cb58b78-cpwpv" Oct 31 00:35:23.597347 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:23.596913 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57cb58b78-cpwpv" event={"ID":"a6079d2f-a234-49c1-9011-57cbba17a01f","Type":"ContainerDied","Data":"1c2cf4b7331070a06fa318a7956a0f586b56ca8454ad84604cc4f84138ec088f"} Oct 31 00:35:23.597347 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:23.596932 4519 scope.go:122] "RemoveContainer" containerID="4c4aebff9a822cda9734112b7bafa7706cf2463687ce426d88f9c3ea2630a557" Oct 31 00:35:23.605845 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:23.605825 4519 scope.go:122] "RemoveContainer" containerID="4c4aebff9a822cda9734112b7bafa7706cf2463687ce426d88f9c3ea2630a557" Oct 31 00:35:23.606146 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:35:23.606128 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c4aebff9a822cda9734112b7bafa7706cf2463687ce426d88f9c3ea2630a557\": container with ID starting with 4c4aebff9a822cda9734112b7bafa7706cf2463687ce426d88f9c3ea2630a557 not found: ID does not exist" containerID="4c4aebff9a822cda9734112b7bafa7706cf2463687ce426d88f9c3ea2630a557" Oct 31 00:35:23.606189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:23.606155 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c4aebff9a822cda9734112b7bafa7706cf2463687ce426d88f9c3ea2630a557"} err="failed to get container status \"4c4aebff9a822cda9734112b7bafa7706cf2463687ce426d88f9c3ea2630a557\": rpc error: code = NotFound desc = could not find container \"4c4aebff9a822cda9734112b7bafa7706cf2463687ce426d88f9c3ea2630a557\": container with ID starting with 4c4aebff9a822cda9734112b7bafa7706cf2463687ce426d88f9c3ea2630a557 not found: ID does not exist" Oct 31 00:35:23.619333 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:23.619290 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57cb58b78-cpwpv"] Oct 31 00:35:23.623013 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:23.622978 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57cb58b78-cpwpv"] Oct 31 00:35:24.099428 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:24.099399 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6079d2f-a234-49c1-9011-57cbba17a01f" path="/var/lib/kubelet/pods/a6079d2f-a234-49c1-9011-57cbba17a01f/volumes" Oct 31 00:35:33.222115 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:33.222076 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:33.317445 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:33.317411 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:36.174781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.174702 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-87554bc7c-dpfbs"] Oct 31 00:35:36.175162 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.175009 4519 cpu_manager.go:414] "containerMap: removing container" podUID="a6079d2f-a234-49c1-9011-57cbba17a01f" containerName="console" Oct 31 00:35:36.175162 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.175021 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6079d2f-a234-49c1-9011-57cbba17a01f" containerName="console" Oct 31 00:35:36.175162 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.175063 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6079d2f-a234-49c1-9011-57cbba17a01f" containerName="console" Oct 31 00:35:36.177800 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.177784 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.189375 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.189350 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87554bc7c-dpfbs"] Oct 31 00:35:36.243181 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.243147 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:36.309240 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.309202 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75b4h\" (UniqueName: \"kubernetes.io/projected/dd367377-297c-4664-8df8-270c8f55d3b9-kube-api-access-75b4h\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.309403 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.309260 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd367377-297c-4664-8df8-270c8f55d3b9-console-oauth-config\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.309403 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.309278 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd367377-297c-4664-8df8-270c8f55d3b9-tmp\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.309403 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.309354 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd367377-297c-4664-8df8-270c8f55d3b9-oauth-serving-cert\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.309501 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.309405 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd367377-297c-4664-8df8-270c8f55d3b9-console-serving-cert\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.309501 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.309426 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd367377-297c-4664-8df8-270c8f55d3b9-service-ca\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.309501 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.309458 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd367377-297c-4664-8df8-270c8f55d3b9-console-config\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.309596 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.309521 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd367377-297c-4664-8df8-270c8f55d3b9-trusted-ca-bundle\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.315572 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.315546 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:36.409975 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.409935 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd367377-297c-4664-8df8-270c8f55d3b9-console-oauth-config\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.409975 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.409971 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd367377-297c-4664-8df8-270c8f55d3b9-tmp\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.410240 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.409990 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd367377-297c-4664-8df8-270c8f55d3b9-oauth-serving-cert\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.410240 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.410154 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd367377-297c-4664-8df8-270c8f55d3b9-console-serving-cert\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.410240 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.410199 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd367377-297c-4664-8df8-270c8f55d3b9-service-ca\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.410393 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.410257 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd367377-297c-4664-8df8-270c8f55d3b9-console-config\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.410393 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.410288 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd367377-297c-4664-8df8-270c8f55d3b9-trusted-ca-bundle\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.410393 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.410326 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75b4h\" (UniqueName: \"kubernetes.io/projected/dd367377-297c-4664-8df8-270c8f55d3b9-kube-api-access-75b4h\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.410831 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.410804 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd367377-297c-4664-8df8-270c8f55d3b9-oauth-serving-cert\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.410965 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.410941 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd367377-297c-4664-8df8-270c8f55d3b9-console-config\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.411067 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.410968 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd367377-297c-4664-8df8-270c8f55d3b9-service-ca\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.411067 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.411057 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd367377-297c-4664-8df8-270c8f55d3b9-tmp\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.411296 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.411273 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd367377-297c-4664-8df8-270c8f55d3b9-trusted-ca-bundle\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.412595 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.412571 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd367377-297c-4664-8df8-270c8f55d3b9-console-serving-cert\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.412717 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.412701 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd367377-297c-4664-8df8-270c8f55d3b9-console-oauth-config\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.418732 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.418710 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75b4h\" (UniqueName: \"kubernetes.io/projected/dd367377-297c-4664-8df8-270c8f55d3b9-kube-api-access-75b4h\") pod \"console-87554bc7c-dpfbs\" (UID: \"dd367377-297c-4664-8df8-270c8f55d3b9\") " pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.489908 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.489844 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:36.764067 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.764043 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-87554bc7c-dpfbs"] Oct 31 00:35:36.766428 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:35:36.766399 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd367377_297c_4664_8df8_270c8f55d3b9.slice/crio-936eb1a1a4a70ab04acaa61e27d503d05e56a810629e4a7bcbcb8dd0e00d4c97 WatchSource:0}: Error finding container 936eb1a1a4a70ab04acaa61e27d503d05e56a810629e4a7bcbcb8dd0e00d4c97: Status 404 returned error can't find the container with id 936eb1a1a4a70ab04acaa61e27d503d05e56a810629e4a7bcbcb8dd0e00d4c97 Oct 31 00:35:36.852348 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.852316 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:36.925360 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:36.925327 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:37.645668 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:37.645623 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87554bc7c-dpfbs" event={"ID":"dd367377-297c-4664-8df8-270c8f55d3b9","Type":"ContainerStarted","Data":"0af2e816e3eb6f9879760243db21ff211bd12ff5bc10c4703b55ac99488d7c2e"} Oct 31 00:35:37.646060 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:37.645687 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-87554bc7c-dpfbs" event={"ID":"dd367377-297c-4664-8df8-270c8f55d3b9","Type":"ContainerStarted","Data":"936eb1a1a4a70ab04acaa61e27d503d05e56a810629e4a7bcbcb8dd0e00d4c97"} Oct 31 00:35:37.666551 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:37.666501 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-87554bc7c-dpfbs" podStartSLOduration=1.666488411 podStartE2EDuration="1.666488411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:37.664334446 +0000 UTC m=+133.693514882" watchObservedRunningTime="2026-10-31 00:35:37.666488411 +0000 UTC m=+133.695668856" Oct 31 00:35:45.928128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:45.928091 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:45.999378 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:45.999338 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:46.490331 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:46.490295 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:46.490502 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:46.490368 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:46.495162 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:46.495142 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:46.677871 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:46.677841 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-87554bc7c-dpfbs" Oct 31 00:35:46.726612 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:46.726577 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d68bbc5f4-4m6gx"] Oct 31 00:35:47.541379 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:35:47.541338 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:35:51.659631 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:51.659599 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:51.758152 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:51.758121 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:51.858847 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:51.858813 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:54.713739 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:35:54.713700 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:09.857670 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:09.857618 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:10.750303 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:10.750269 4519 generic.go:356] "Generic (PLEG): container finished" podID="623173be-674f-4b4c-8920-d8b1f213666b" containerID="c517725c8286577dba9f7bae952f16965121576a654f99fcfea5477cc4c36fd2" exitCode=0 Oct 31 00:36:10.750479 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:10.750353 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" event={"ID":"623173be-674f-4b4c-8920-d8b1f213666b","Type":"ContainerDied","Data":"c517725c8286577dba9f7bae952f16965121576a654f99fcfea5477cc4c36fd2"} Oct 31 00:36:11.745953 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:11.745917 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6d68bbc5f4-4m6gx" podUID="1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed" containerName="console" containerID="cri-o://0bbb4ac4986aea14bfb9d4c26fd5e353f37d800270353c05824cfac9c7f5d754" gracePeriod=15 Oct 31 00:36:11.885814 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:11.885789 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" Oct 31 00:36:11.952180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:11.952139 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/623173be-674f-4b4c-8920-d8b1f213666b-kube-api-access-d4c46\" (UniqueName: \"kubernetes.io/projected/623173be-674f-4b4c-8920-d8b1f213666b-kube-api-access-d4c46\") pod \"623173be-674f-4b4c-8920-d8b1f213666b\" (UID: \"623173be-674f-4b4c-8920-d8b1f213666b\") " Oct 31 00:36:11.952372 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:11.952205 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623173be-674f-4b4c-8920-d8b1f213666b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/623173be-674f-4b4c-8920-d8b1f213666b-archives-path\") pod \"623173be-674f-4b4c-8920-d8b1f213666b\" (UID: \"623173be-674f-4b4c-8920-d8b1f213666b\") " Oct 31 00:36:11.952372 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:11.952275 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/623173be-674f-4b4c-8920-d8b1f213666b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/623173be-674f-4b4c-8920-d8b1f213666b-trusted-ca-bundle\") pod \"623173be-674f-4b4c-8920-d8b1f213666b\" (UID: \"623173be-674f-4b4c-8920-d8b1f213666b\") " Oct 31 00:36:11.952372 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:11.952308 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/623173be-674f-4b4c-8920-d8b1f213666b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/623173be-674f-4b4c-8920-d8b1f213666b-service-ca-bundle\") pod \"623173be-674f-4b4c-8920-d8b1f213666b\" (UID: \"623173be-674f-4b4c-8920-d8b1f213666b\") " Oct 31 00:36:11.952549 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:11.952519 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623173be-674f-4b4c-8920-d8b1f213666b-archives-path" pod "623173be-674f-4b4c-8920-d8b1f213666b" (UID: "623173be-674f-4b4c-8920-d8b1f213666b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.952694 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:11.952674 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623173be-674f-4b4c-8920-d8b1f213666b-trusted-ca-bundle" pod "623173be-674f-4b4c-8920-d8b1f213666b" (UID: "623173be-674f-4b4c-8920-d8b1f213666b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:11.952767 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:11.952709 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623173be-674f-4b4c-8920-d8b1f213666b-service-ca-bundle" pod "623173be-674f-4b4c-8920-d8b1f213666b" (UID: "623173be-674f-4b4c-8920-d8b1f213666b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:11.954285 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:11.954261 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623173be-674f-4b4c-8920-d8b1f213666b-kube-api-access-d4c46" pod "623173be-674f-4b4c-8920-d8b1f213666b" (UID: "623173be-674f-4b4c-8920-d8b1f213666b"). InnerVolumeSpecName "kube-api-access-d4c46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.994120 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:11.994088 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:36:12.053113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053008 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-trusted-ca-bundle\") pod \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " Oct 31 00:36:12.053113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053057 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-oauth-config\") pod \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " Oct 31 00:36:12.053113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053094 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-service-ca\") pod \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " Oct 31 00:36:12.053113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053115 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-tmp\") pod \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " Oct 31 00:36:12.053449 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053162 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-oauth-serving-cert\") pod \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " Oct 31 00:36:12.053449 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053187 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-config\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-config\") pod \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " Oct 31 00:36:12.053449 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053213 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-kube-api-access-zbgm2\" (UniqueName: \"kubernetes.io/projected/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-kube-api-access-zbgm2\") pod \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " Oct 31 00:36:12.053449 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053281 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-serving-cert\") pod \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\" (UID: \"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed\") " Oct 31 00:36:12.053678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053451 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-tmp" pod "1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed" (UID: "1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:12.053678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053510 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-service-ca" pod "1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed" (UID: "1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:12.053678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053582 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-trusted-ca-bundle" pod "1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed" (UID: "1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:12.053678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053600 4519 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/623173be-674f-4b4c-8920-d8b1f213666b-archives-path\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.053678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053622 4519 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/623173be-674f-4b4c-8920-d8b1f213666b-trusted-ca-bundle\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.053678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053663 4519 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/623173be-674f-4b4c-8920-d8b1f213666b-service-ca-bundle\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.053678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053677 4519 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-service-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.053980 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053689 4519 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-tmp\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.053980 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053704 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4c46\" (UniqueName: \"kubernetes.io/projected/623173be-674f-4b4c-8920-d8b1f213666b-kube-api-access-d4c46\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.053980 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053824 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-oauth-serving-cert" pod "1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed" (UID: "1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:12.053980 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.053866 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-config" pod "1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed" (UID: "1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:12.055281 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.055258 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-oauth-config" pod "1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed" (UID: "1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:12.055483 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.055463 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-kube-api-access-zbgm2" pod "1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed" (UID: "1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed"). InnerVolumeSpecName "kube-api-access-zbgm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:12.055519 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.055469 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-serving-cert" pod "1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed" (UID: "1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:12.154817 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.154776 4519 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-oauth-serving-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.154817 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.154811 4519 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-config\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.154817 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.154821 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbgm2\" (UniqueName: \"kubernetes.io/projected/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-kube-api-access-zbgm2\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.154817 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.154830 4519 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-serving-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.155098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.154839 4519 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-trusted-ca-bundle\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.155098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.154849 4519 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed-console-oauth-config\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.758899 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.758875 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" Oct 31 00:36:12.759403 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.758919 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fg9fp-xtjlp" event={"ID":"623173be-674f-4b4c-8920-d8b1f213666b","Type":"ContainerDied","Data":"0b2de6e6c23422033c2f4de7caefceac127006bf288f28337df8b76008164ef3"} Oct 31 00:36:12.759403 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.758975 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b2de6e6c23422033c2f4de7caefceac127006bf288f28337df8b76008164ef3" Oct 31 00:36:12.760349 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.760323 4519 generic.go:356] "Generic (PLEG): container finished" podID="1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed" containerID="0bbb4ac4986aea14bfb9d4c26fd5e353f37d800270353c05824cfac9c7f5d754" exitCode=0 Oct 31 00:36:12.760467 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.760364 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d68bbc5f4-4m6gx" Oct 31 00:36:12.760467 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.760393 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d68bbc5f4-4m6gx" event={"ID":"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed","Type":"ContainerDied","Data":"0bbb4ac4986aea14bfb9d4c26fd5e353f37d800270353c05824cfac9c7f5d754"} Oct 31 00:36:12.760467 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.760437 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d68bbc5f4-4m6gx" event={"ID":"1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed","Type":"ContainerDied","Data":"5cca55460ad9752386d5ac36c9d8b492851a8c5118da0ce3f370ff82ef013693"} Oct 31 00:36:12.760467 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.760463 4519 scope.go:122] "RemoveContainer" containerID="0bbb4ac4986aea14bfb9d4c26fd5e353f37d800270353c05824cfac9c7f5d754" Oct 31 00:36:12.768436 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.768416 4519 scope.go:122] "RemoveContainer" containerID="0bbb4ac4986aea14bfb9d4c26fd5e353f37d800270353c05824cfac9c7f5d754" Oct 31 00:36:12.768693 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:36:12.768676 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bbb4ac4986aea14bfb9d4c26fd5e353f37d800270353c05824cfac9c7f5d754\": container with ID starting with 0bbb4ac4986aea14bfb9d4c26fd5e353f37d800270353c05824cfac9c7f5d754 not found: ID does not exist" containerID="0bbb4ac4986aea14bfb9d4c26fd5e353f37d800270353c05824cfac9c7f5d754" Oct 31 00:36:12.768739 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.768701 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bbb4ac4986aea14bfb9d4c26fd5e353f37d800270353c05824cfac9c7f5d754"} err="failed to get container status \"0bbb4ac4986aea14bfb9d4c26fd5e353f37d800270353c05824cfac9c7f5d754\": rpc error: code = NotFound desc = could not find container \"0bbb4ac4986aea14bfb9d4c26fd5e353f37d800270353c05824cfac9c7f5d754\": container with ID starting with 0bbb4ac4986aea14bfb9d4c26fd5e353f37d800270353c05824cfac9c7f5d754 not found: ID does not exist" Oct 31 00:36:12.781184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.781161 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d68bbc5f4-4m6gx"] Oct 31 00:36:12.783846 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:12.783824 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d68bbc5f4-4m6gx"] Oct 31 00:36:13.053897 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:13.053803 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:13.120151 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:13.120122 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:14.099442 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:14.099406 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed" path="/var/lib/kubelet/pods/1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed/volumes" Oct 31 00:36:17.539370 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:36:17.539327 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:36:46.717001 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:46.716963 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:46.783298 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:36:46.783267 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:47.541108 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:36:47.541067 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:37:08.524977 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:08.524897 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:08.615976 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:08.615942 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:09.903033 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:09.902998 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:17.539967 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:37:17.539930 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:37:21.255778 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:21.255746 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:21.370013 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:21.369979 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:21.499629 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:21.499593 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:22.673792 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.673762 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn"] Oct 31 00:37:22.674184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.674042 4519 cpu_manager.go:414] "containerMap: removing container" podUID="1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed" containerName="console" Oct 31 00:37:22.674184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.674054 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed" containerName="console" Oct 31 00:37:22.674184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.674067 4519 cpu_manager.go:414] "containerMap: removing container" podUID="623173be-674f-4b4c-8920-d8b1f213666b" containerName="insights-gathering" Oct 31 00:37:22.674184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.674072 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="623173be-674f-4b4c-8920-d8b1f213666b" containerName="insights-gathering" Oct 31 00:37:22.674184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.674128 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b4e3f3f-d16d-49e2-a862-f5be9e43d4ed" containerName="console" Oct 31 00:37:22.674184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.674137 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="623173be-674f-4b4c-8920-d8b1f213666b" containerName="insights-gathering" Oct 31 00:37:22.676961 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.676944 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.681452 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.681425 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 00:37:22.681452 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.681443 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 00:37:22.681673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.681535 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 00:37:22.687200 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.687177 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn"] Oct 31 00:37:22.690227 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.690193 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:22.691995 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.691972 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-srs6x\"" Oct 31 00:37:22.691995 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.691974 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 00:37:22.692739 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.692716 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 00:37:22.692909 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.692890 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 00:37:22.774833 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.774795 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71c13e9-da5c-44e2-9bb5-e40418b1544d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.774833 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.774836 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.775048 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.774923 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.775048 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.774961 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4zxz\" (UniqueName: \"kubernetes.io/projected/f71c13e9-da5c-44e2-9bb5-e40418b1544d-kube-api-access-j4zxz\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.775048 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.774981 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.775048 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.775029 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.775194 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.775101 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.785316 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.785290 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:22.875996 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.875951 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71c13e9-da5c-44e2-9bb5-e40418b1544d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.876184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.876003 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.876184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.876040 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.876184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.876064 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4zxz\" (UniqueName: \"kubernetes.io/projected/f71c13e9-da5c-44e2-9bb5-e40418b1544d-kube-api-access-j4zxz\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.876184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.876086 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.876380 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.876228 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.876380 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.876321 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.877213 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.877128 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.882773 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.878778 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71c13e9-da5c-44e2-9bb5-e40418b1544d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.882773 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.879362 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.882773 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.879406 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.882773 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.879862 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.888179 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.888155 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.888551 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.888532 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4zxz\" (UniqueName: \"kubernetes.io/projected/f71c13e9-da5c-44e2-9bb5-e40418b1544d-kube-api-access-j4zxz\") pod \"istiod-openshift-gateway-7876bfcb74-dtdbn\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:22.988994 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:22.988954 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:23.263217 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:23.263076 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn"] Oct 31 00:37:23.266105 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:37:23.266074 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71c13e9_da5c_44e2_9bb5_e40418b1544d.slice/crio-0468c3ce398114ccbc5d8f6e06c72bf7bc5673efd30542ac7d83841ca2d8588e WatchSource:0}: Error finding container 0468c3ce398114ccbc5d8f6e06c72bf7bc5673efd30542ac7d83841ca2d8588e: Status 404 returned error can't find the container with id 0468c3ce398114ccbc5d8f6e06c72bf7bc5673efd30542ac7d83841ca2d8588e Oct 31 00:37:23.625835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:23.625729 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:24.001723 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:24.001686 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" event={"ID":"f71c13e9-da5c-44e2-9bb5-e40418b1544d","Type":"ContainerStarted","Data":"0468c3ce398114ccbc5d8f6e06c72bf7bc5673efd30542ac7d83841ca2d8588e"} Oct 31 00:37:26.404964 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:26.404929 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:37:26.405251 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:26.404992 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:37:26.846149 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:26.846106 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:26.956867 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:26.956836 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:27.015496 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:27.015461 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" event={"ID":"f71c13e9-da5c-44e2-9bb5-e40418b1544d","Type":"ContainerStarted","Data":"6001200c2de4903b65625863aba623fe19b20af3074b43043cda6d6e432c50d6"} Oct 31 00:37:27.015692 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:27.015660 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:27.017331 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:27.017282 4519 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-dtdbn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:37:27.017469 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:27.017370 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" podUID="f71c13e9-da5c-44e2-9bb5-e40418b1544d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:37:27.047879 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:27.047828 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" podStartSLOduration=1.912167119 podStartE2EDuration="5.04781365s" totalImagesPullingTime="3.135646531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:27.047053881 +0000 UTC m=+243.076234271" watchObservedRunningTime="2026-10-31 00:37:27.04781365 +0000 UTC m=+243.076994040" Oct 31 00:37:27.622895 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:27.622864 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:27.702548 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:27.702515 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:27.807861 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:27.807830 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:27.884510 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:27.884433 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:28.019260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:28.019232 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:37:30.311181 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:30.310940 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:47.540467 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:37:47.540419 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:37:56.884421 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:56.884382 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:57.504201 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:37:57.504158 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:05.866947 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:05.866888 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:06.849445 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:06.849397 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:07.860961 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:07.860929 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:09.939584 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:09.939549 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:12.143823 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:12.143786 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:12.344245 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:12.344210 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:13.408430 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.408395 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:13.579749 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.579712 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn"] Oct 31 00:38:13.583325 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.583295 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.614584 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.614551 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn"] Oct 31 00:38:13.647835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.647800 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd5hp\" (UniqueName: \"kubernetes.io/projected/e9cd99cf-c934-4418-b568-e254d3461a7f-kube-api-access-qd5hp\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.648052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.647869 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e9cd99cf-c934-4418-b568-e254d3461a7f-metrics-cert\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.648052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.647984 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9cd99cf-c934-4418-b568-e254d3461a7f-tmp\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.648052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.648021 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e9cd99cf-c934-4418-b568-e254d3461a7f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.648189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.648064 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e9cd99cf-c934-4418-b568-e254d3461a7f-manifests\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.648189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.648108 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9cd99cf-c934-4418-b568-e254d3461a7f-webhook-cert\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.664387 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.664298 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-j6mcx\"" Oct 31 00:38:13.664387 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.664297 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 00:38:13.664387 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.664277 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 00:38:13.664387 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.664344 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 00:38:13.665059 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.665040 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 00:38:13.665142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.665073 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 00:38:13.680222 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.680191 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:13.749204 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.749158 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e9cd99cf-c934-4418-b568-e254d3461a7f-metrics-cert\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.749406 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.749214 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9cd99cf-c934-4418-b568-e254d3461a7f-tmp\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.749406 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.749249 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e9cd99cf-c934-4418-b568-e254d3461a7f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.749406 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.749271 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e9cd99cf-c934-4418-b568-e254d3461a7f-manifests\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.749406 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.749337 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9cd99cf-c934-4418-b568-e254d3461a7f-webhook-cert\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.749406 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.749402 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd5hp\" (UniqueName: \"kubernetes.io/projected/e9cd99cf-c934-4418-b568-e254d3461a7f-kube-api-access-qd5hp\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.749766 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.749740 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e9cd99cf-c934-4418-b568-e254d3461a7f-manifests\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.751740 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.751714 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9cd99cf-c934-4418-b568-e254d3461a7f-tmp\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.751875 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.751861 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9cd99cf-c934-4418-b568-e254d3461a7f-webhook-cert\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.751875 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.751870 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e9cd99cf-c934-4418-b568-e254d3461a7f-metrics-cert\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.752130 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.752108 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e9cd99cf-c934-4418-b568-e254d3461a7f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.768033 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.768001 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd5hp\" (UniqueName: \"kubernetes.io/projected/e9cd99cf-c934-4418-b568-e254d3461a7f-kube-api-access-qd5hp\") pod \"opendatahub-operator-controller-manager-54459f848c-m9zdn\" (UID: \"e9cd99cf-c934-4418-b568-e254d3461a7f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.783744 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.783708 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:13.876244 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.876207 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:13.898500 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.898465 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:13.976081 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:13.976049 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:14.080691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:14.080609 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:14.194979 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:14.194942 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:14.211429 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:14.211397 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn"] Oct 31 00:38:14.212770 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:38:14.212744 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cd99cf_c934_4418_b568_e254d3461a7f.slice/crio-115de7faa7b960196ff99bc19fb3aa225f9c45f2a85b8aff6416653f1109b454 WatchSource:0}: Error finding container 115de7faa7b960196ff99bc19fb3aa225f9c45f2a85b8aff6416653f1109b454: Status 404 returned error can't find the container with id 115de7faa7b960196ff99bc19fb3aa225f9c45f2a85b8aff6416653f1109b454 Oct 31 00:38:14.279121 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:14.279091 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:14.379562 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:14.379532 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:15.173101 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:15.173065 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" event={"ID":"e9cd99cf-c934-4418-b568-e254d3461a7f","Type":"ContainerStarted","Data":"115de7faa7b960196ff99bc19fb3aa225f9c45f2a85b8aff6416653f1109b454"} Oct 31 00:38:16.461835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:16.461804 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:17.097076 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:17.097040 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:17.543836 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:38:17.543792 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:38:18.185721 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:18.185683 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" event={"ID":"e9cd99cf-c934-4418-b568-e254d3461a7f","Type":"ContainerStarted","Data":"0b3e299f6b5754b027d2b72098f1480ba586b6c8926a7fe604dc982a3038f10c"} Oct 31 00:38:19.190909 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:19.190875 4519 generic.go:356] "Generic (PLEG): container finished" podID="e9cd99cf-c934-4418-b568-e254d3461a7f" containerID="0b3e299f6b5754b027d2b72098f1480ba586b6c8926a7fe604dc982a3038f10c" exitCode=0 Oct 31 00:38:19.191298 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:19.190958 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" event={"ID":"e9cd99cf-c934-4418-b568-e254d3461a7f","Type":"ContainerDied","Data":"0b3e299f6b5754b027d2b72098f1480ba586b6c8926a7fe604dc982a3038f10c"} Oct 31 00:38:19.658275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:19.658248 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:20.005090 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:20.005049 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:20.196795 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:20.196763 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" event={"ID":"e9cd99cf-c934-4418-b568-e254d3461a7f","Type":"ContainerStarted","Data":"bc5990aec6071c19f205167d109a758dfb732b9fdbb34955a9ce1d082040e186"} Oct 31 00:38:20.197192 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:20.196893 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:20.241128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:20.241078 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" podStartSLOduration=2.605977996 podStartE2EDuration="7.241064243s" totalImagesPullingTime="4.194963007s" totalInitContainerRuntime="440.12324ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:20.23962312 +0000 UTC m=+296.268803505" watchObservedRunningTime="2026-10-31 00:38:20.241064243 +0000 UTC m=+296.270244632" Oct 31 00:38:20.855494 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:20.855458 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:20.951670 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:20.951618 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:21.675423 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:21.675397 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:22.408724 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:22.408683 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:22.522017 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:22.521981 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:22.904890 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:22.904859 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:24.321901 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:24.321858 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:24.546088 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:24.546054 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:24.900145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:24.900097 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:25.540111 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:25.540075 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:25.894080 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:25.893996 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:25.998764 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:25.998726 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:27.303022 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:27.302990 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:27.432896 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:27.432864 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:30.532491 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:30.532453 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:30.636592 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:30.636553 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:31.203450 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:31.203416 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54459f848c-m9zdn" Oct 31 00:38:31.366942 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:31.366909 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:31.501732 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:31.501698 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:31.613596 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:31.613566 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:32.311098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:32.311063 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:32.411484 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:32.411452 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:32.510607 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:32.510578 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:33.181530 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:33.181497 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:33.753796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:33.753763 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:40.097737 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:40.097647 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:40.190795 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:40.190759 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:40.679087 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:40.679058 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:40.778588 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:40.778560 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:42.526239 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:42.526202 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:43.596840 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:43.596799 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:43.723290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:43.723262 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:43.954068 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:43.952991 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b544748f5-zkdsq"] Oct 31 00:38:43.956327 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:43.956291 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b544748f5-zkdsq" Oct 31 00:38:43.964493 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:43.964471 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gddhf\"" Oct 31 00:38:43.964602 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:43.964497 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 00:38:43.964602 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:43.964540 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 00:38:43.965338 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:43.965300 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:43.968258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:43.968236 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b544748f5-zkdsq"] Oct 31 00:38:43.989898 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:43.989867 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7213fb5-28f3-41c1-bd33-4e922f88120d-tls-certs\") pod \"kube-auth-proxy-b544748f5-zkdsq\" (UID: \"d7213fb5-28f3-41c1-bd33-4e922f88120d\") " pod="openshift-ingress/kube-auth-proxy-b544748f5-zkdsq" Oct 31 00:38:43.990068 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:43.989916 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzxcc\" (UniqueName: \"kubernetes.io/projected/d7213fb5-28f3-41c1-bd33-4e922f88120d-kube-api-access-lzxcc\") pod \"kube-auth-proxy-b544748f5-zkdsq\" (UID: \"d7213fb5-28f3-41c1-bd33-4e922f88120d\") " pod="openshift-ingress/kube-auth-proxy-b544748f5-zkdsq" Oct 31 00:38:43.990068 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:43.989949 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7213fb5-28f3-41c1-bd33-4e922f88120d-tmp\") pod \"kube-auth-proxy-b544748f5-zkdsq\" (UID: \"d7213fb5-28f3-41c1-bd33-4e922f88120d\") " pod="openshift-ingress/kube-auth-proxy-b544748f5-zkdsq" Oct 31 00:38:44.051929 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:44.051889 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:44.091085 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:44.091051 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7213fb5-28f3-41c1-bd33-4e922f88120d-tls-certs\") pod \"kube-auth-proxy-b544748f5-zkdsq\" (UID: \"d7213fb5-28f3-41c1-bd33-4e922f88120d\") " pod="openshift-ingress/kube-auth-proxy-b544748f5-zkdsq" Oct 31 00:38:44.091269 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:44.091101 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzxcc\" (UniqueName: \"kubernetes.io/projected/d7213fb5-28f3-41c1-bd33-4e922f88120d-kube-api-access-lzxcc\") pod \"kube-auth-proxy-b544748f5-zkdsq\" (UID: \"d7213fb5-28f3-41c1-bd33-4e922f88120d\") " pod="openshift-ingress/kube-auth-proxy-b544748f5-zkdsq" Oct 31 00:38:44.091269 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:44.091124 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7213fb5-28f3-41c1-bd33-4e922f88120d-tmp\") pod \"kube-auth-proxy-b544748f5-zkdsq\" (UID: \"d7213fb5-28f3-41c1-bd33-4e922f88120d\") " pod="openshift-ingress/kube-auth-proxy-b544748f5-zkdsq" Oct 31 00:38:44.093360 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:44.093329 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7213fb5-28f3-41c1-bd33-4e922f88120d-tmp\") pod \"kube-auth-proxy-b544748f5-zkdsq\" (UID: \"d7213fb5-28f3-41c1-bd33-4e922f88120d\") " pod="openshift-ingress/kube-auth-proxy-b544748f5-zkdsq" Oct 31 00:38:44.093611 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:44.093592 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7213fb5-28f3-41c1-bd33-4e922f88120d-tls-certs\") pod \"kube-auth-proxy-b544748f5-zkdsq\" (UID: \"d7213fb5-28f3-41c1-bd33-4e922f88120d\") " pod="openshift-ingress/kube-auth-proxy-b544748f5-zkdsq" Oct 31 00:38:44.110702 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:44.110673 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzxcc\" (UniqueName: \"kubernetes.io/projected/d7213fb5-28f3-41c1-bd33-4e922f88120d-kube-api-access-lzxcc\") pod \"kube-auth-proxy-b544748f5-zkdsq\" (UID: \"d7213fb5-28f3-41c1-bd33-4e922f88120d\") " pod="openshift-ingress/kube-auth-proxy-b544748f5-zkdsq" Oct 31 00:38:44.148610 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:44.148582 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:44.255383 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:44.255345 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:44.272353 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:44.272324 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b544748f5-zkdsq" Oct 31 00:38:44.545278 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:44.545124 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b544748f5-zkdsq"] Oct 31 00:38:44.548188 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:38:44.548159 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7213fb5_28f3_41c1_bd33_4e922f88120d.slice/crio-b47c489cd9ec1cf5b2a39176133b1e0b1f4ec116ee429f4ac2289a77902b31ea WatchSource:0}: Error finding container b47c489cd9ec1cf5b2a39176133b1e0b1f4ec116ee429f4ac2289a77902b31ea: Status 404 returned error can't find the container with id b47c489cd9ec1cf5b2a39176133b1e0b1f4ec116ee429f4ac2289a77902b31ea Oct 31 00:38:44.550050 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:44.550033 4519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:38:44.715234 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:44.715200 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:44.805609 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:44.805531 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:44.908008 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:44.907975 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:45.104138 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:45.104066 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:45.195419 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:45.195382 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:45.293198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:45.293166 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b544748f5-zkdsq" event={"ID":"d7213fb5-28f3-41c1-bd33-4e922f88120d","Type":"ContainerStarted","Data":"b47c489cd9ec1cf5b2a39176133b1e0b1f4ec116ee429f4ac2289a77902b31ea"} Oct 31 00:38:45.682531 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:45.682495 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:45.789899 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:45.789868 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:45.897864 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:45.897833 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h"] Oct 31 00:38:45.901061 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:45.901040 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" Oct 31 00:38:45.904872 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:45.904841 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-szwdn\"" Oct 31 00:38:45.911649 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:45.911619 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h"] Oct 31 00:38:45.940874 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:45.940776 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 00:38:45.940874 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:45.940783 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 00:38:45.949118 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:45.949078 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 31 00:38:45.976690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:45.976658 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-msz8z"] Oct 31 00:38:45.982680 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:45.982651 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-msz8z" Oct 31 00:38:45.986697 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:45.986624 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-msz8z"] Oct 31 00:38:45.988094 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:45.988068 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 00:38:45.988225 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:45.988085 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 00:38:46.001989 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.001966 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-czhr7\"" Oct 31 00:38:46.005561 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.005533 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcvcb\" (UniqueName: \"kubernetes.io/projected/43e494ff-c9a1-4707-8f27-efd46b218d97-kube-api-access-fcvcb\") pod \"kserve-module-controller-manager-7566cf5fc8-8kw7h\" (UID: \"43e494ff-c9a1-4707-8f27-efd46b218d97\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" Oct 31 00:38:46.005723 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.005622 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xq6m\" (UniqueName: \"kubernetes.io/projected/75f2bea7-a08e-4c6f-a405-b64c48fb85cc-kube-api-access-2xq6m\") pod \"authorino-56c6c9ddd5-msz8z\" (UID: \"75f2bea7-a08e-4c6f-a405-b64c48fb85cc\") " pod="kuadrant-system/authorino-56c6c9ddd5-msz8z" Oct 31 00:38:46.005723 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.005673 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/43e494ff-c9a1-4707-8f27-efd46b218d97-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8kw7h\" (UID: \"43e494ff-c9a1-4707-8f27-efd46b218d97\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" Oct 31 00:38:46.005840 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.005715 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43e494ff-c9a1-4707-8f27-efd46b218d97-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8kw7h\" (UID: \"43e494ff-c9a1-4707-8f27-efd46b218d97\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" Oct 31 00:38:46.005896 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.005849 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43e494ff-c9a1-4707-8f27-efd46b218d97-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8kw7h\" (UID: \"43e494ff-c9a1-4707-8f27-efd46b218d97\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" Oct 31 00:38:46.005955 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.005891 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:46.106496 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.106452 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcvcb\" (UniqueName: \"kubernetes.io/projected/43e494ff-c9a1-4707-8f27-efd46b218d97-kube-api-access-fcvcb\") pod \"kserve-module-controller-manager-7566cf5fc8-8kw7h\" (UID: \"43e494ff-c9a1-4707-8f27-efd46b218d97\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" Oct 31 00:38:46.106496 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.106500 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xq6m\" (UniqueName: \"kubernetes.io/projected/75f2bea7-a08e-4c6f-a405-b64c48fb85cc-kube-api-access-2xq6m\") pod \"authorino-56c6c9ddd5-msz8z\" (UID: \"75f2bea7-a08e-4c6f-a405-b64c48fb85cc\") " pod="kuadrant-system/authorino-56c6c9ddd5-msz8z" Oct 31 00:38:46.106766 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.106526 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/43e494ff-c9a1-4707-8f27-efd46b218d97-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8kw7h\" (UID: \"43e494ff-c9a1-4707-8f27-efd46b218d97\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" Oct 31 00:38:46.106766 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.106543 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43e494ff-c9a1-4707-8f27-efd46b218d97-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8kw7h\" (UID: \"43e494ff-c9a1-4707-8f27-efd46b218d97\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" Oct 31 00:38:46.106766 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.106570 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43e494ff-c9a1-4707-8f27-efd46b218d97-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8kw7h\" (UID: \"43e494ff-c9a1-4707-8f27-efd46b218d97\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" Oct 31 00:38:46.106951 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.106924 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/43e494ff-c9a1-4707-8f27-efd46b218d97-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8kw7h\" (UID: \"43e494ff-c9a1-4707-8f27-efd46b218d97\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" Oct 31 00:38:46.107075 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.106971 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43e494ff-c9a1-4707-8f27-efd46b218d97-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8kw7h\" (UID: \"43e494ff-c9a1-4707-8f27-efd46b218d97\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" Oct 31 00:38:46.107452 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.107434 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43e494ff-c9a1-4707-8f27-efd46b218d97-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8kw7h\" (UID: \"43e494ff-c9a1-4707-8f27-efd46b218d97\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" Oct 31 00:38:46.115807 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.115774 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xq6m\" (UniqueName: \"kubernetes.io/projected/75f2bea7-a08e-4c6f-a405-b64c48fb85cc-kube-api-access-2xq6m\") pod \"authorino-56c6c9ddd5-msz8z\" (UID: \"75f2bea7-a08e-4c6f-a405-b64c48fb85cc\") " pod="kuadrant-system/authorino-56c6c9ddd5-msz8z" Oct 31 00:38:46.115940 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.115865 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcvcb\" (UniqueName: \"kubernetes.io/projected/43e494ff-c9a1-4707-8f27-efd46b218d97-kube-api-access-fcvcb\") pod \"kserve-module-controller-manager-7566cf5fc8-8kw7h\" (UID: \"43e494ff-c9a1-4707-8f27-efd46b218d97\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" Oct 31 00:38:46.214915 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.214844 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" Oct 31 00:38:46.304047 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.304012 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-msz8z" Oct 31 00:38:46.505014 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.504982 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h"] Oct 31 00:38:46.507486 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:38:46.507449 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e494ff_c9a1_4707_8f27_efd46b218d97.slice/crio-d8a8b3825c083f42e315e31bd71f4075725ec189074a21bf7d691bb0e7b9d69a WatchSource:0}: Error finding container d8a8b3825c083f42e315e31bd71f4075725ec189074a21bf7d691bb0e7b9d69a: Status 404 returned error can't find the container with id d8a8b3825c083f42e315e31bd71f4075725ec189074a21bf7d691bb0e7b9d69a Oct 31 00:38:46.583619 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:46.583581 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-msz8z"] Oct 31 00:38:46.586696 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:38:46.586664 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75f2bea7_a08e_4c6f_a405_b64c48fb85cc.slice/crio-78e5cdc0a46e3b98ee6a383f4d28c2cd93499e3d4be9c74d6154d69f25d95521 WatchSource:0}: Error finding container 78e5cdc0a46e3b98ee6a383f4d28c2cd93499e3d4be9c74d6154d69f25d95521: Status 404 returned error can't find the container with id 78e5cdc0a46e3b98ee6a383f4d28c2cd93499e3d4be9c74d6154d69f25d95521 Oct 31 00:38:47.304571 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:47.304485 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" event={"ID":"43e494ff-c9a1-4707-8f27-efd46b218d97","Type":"ContainerStarted","Data":"d8a8b3825c083f42e315e31bd71f4075725ec189074a21bf7d691bb0e7b9d69a"} Oct 31 00:38:47.306029 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:47.305998 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b544748f5-zkdsq" event={"ID":"d7213fb5-28f3-41c1-bd33-4e922f88120d","Type":"ContainerStarted","Data":"d264c12f89eeb9cf195d8bdb206da8c857cbaba20385000be7a36837b0a7c8cd"} Oct 31 00:38:47.307193 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:47.307165 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-msz8z" event={"ID":"75f2bea7-a08e-4c6f-a405-b64c48fb85cc","Type":"ContainerStarted","Data":"78e5cdc0a46e3b98ee6a383f4d28c2cd93499e3d4be9c74d6154d69f25d95521"} Oct 31 00:38:47.325217 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:47.325013 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b544748f5-zkdsq" podStartSLOduration=1.848734646 podStartE2EDuration="4.324994021s" totalImagesPullingTime="2.476259375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:47.324715147 +0000 UTC m=+323.353895539" watchObservedRunningTime="2026-10-31 00:38:47.324994021 +0000 UTC m=+323.354174413" Oct 31 00:38:47.543676 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:38:47.543625 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:38:48.314094 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:48.314053 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" event={"ID":"43e494ff-c9a1-4707-8f27-efd46b218d97","Type":"ContainerStarted","Data":"c03cbfe9fdc5b78d376b9f04d26e2e6163b9df2d0340a2624f50a8b35efe2f6a"} Oct 31 00:38:48.339477 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:48.339413 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" podStartSLOduration=1.74698299 podStartE2EDuration="3.33939509s" totalImagesPullingTime="1.5924121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:48.33886284 +0000 UTC m=+324.368043233" watchObservedRunningTime="2026-10-31 00:38:48.33939509 +0000 UTC m=+324.368575482" Oct 31 00:38:49.072229 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:49.072190 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:49.138305 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:49.138268 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:49.318147 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:49.318109 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" Oct 31 00:38:50.323914 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:50.323871 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-msz8z" event={"ID":"75f2bea7-a08e-4c6f-a405-b64c48fb85cc","Type":"ContainerStarted","Data":"0f993f5cda9ac1f866d5e2fa363c431d41a6708ba0db878c7523c68317ccf8ff"} Oct 31 00:38:50.340505 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:38:50.340451 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-msz8z" podStartSLOduration=2.202400973 podStartE2EDuration="5.340435107s" totalImagesPullingTime="3.138034134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:50.339923731 +0000 UTC m=+326.369104122" watchObservedRunningTime="2026-10-31 00:38:50.340435107 +0000 UTC m=+326.369615498" Oct 31 00:39:09.989362 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:09.989324 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:10.325106 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:10.325029 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kw7h" Oct 31 00:39:11.291690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:11.291658 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:11.399744 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:11.399694 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:12.130351 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:12.130320 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:17.542411 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:39:17.542361 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:39:18.281568 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:18.281530 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:18.344173 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:18.344137 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:24.107993 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:24.107966 4519 scope.go:122] "RemoveContainer" containerID="f503f1766c023f8399506061e61349cc6e9388be692a0560c0039fe55393c728" Oct 31 00:39:26.749216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:26.749175 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:29.697948 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:29.697900 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:29.763488 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:29.763449 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:38.270064 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.270021 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:38.355019 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.354983 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:38.458549 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.458515 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:38.548108 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.548009 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:38.584199 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.584157 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg"] Oct 31 00:39:38.588693 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.588667 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" Oct 31 00:39:38.589077 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.589039 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v"] Oct 31 00:39:38.593569 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.593227 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg"] Oct 31 00:39:38.593569 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.593342 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.605303 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.605272 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v"] Oct 31 00:39:38.618280 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.618244 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gb4m7\"" Oct 31 00:39:38.624193 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.624162 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 31 00:39:38.645838 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.645807 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:38.654398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.654363 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t54kt\" (UniqueName: \"kubernetes.io/projected/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-kube-api-access-t54kt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.654568 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.654410 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.654568 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.654477 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.654568 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.654525 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0c95901-6cf5-4cd4-9e0f-4c891aa1d591-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sdqcg\" (UID: \"e0c95901-6cf5-4cd4-9e0f-4c891aa1d591\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" Oct 31 00:39:38.654568 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.654544 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfv9r\" (UniqueName: \"kubernetes.io/projected/e0c95901-6cf5-4cd4-9e0f-4c891aa1d591-kube-api-access-pfv9r\") pod \"llmisvc-controller-manager-869d5dcc5d-sdqcg\" (UID: \"e0c95901-6cf5-4cd4-9e0f-4c891aa1d591\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" Oct 31 00:39:38.654568 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.654561 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.654863 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.654575 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.654863 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.654718 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.654863 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.654774 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.725830 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.725795 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:38.756053 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.756013 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.756053 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.756052 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.756297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.756089 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t54kt\" (UniqueName: \"kubernetes.io/projected/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-kube-api-access-t54kt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.756297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.756113 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.756297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.756133 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.756297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.756277 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0c95901-6cf5-4cd4-9e0f-4c891aa1d591-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sdqcg\" (UID: \"e0c95901-6cf5-4cd4-9e0f-4c891aa1d591\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" Oct 31 00:39:38.756506 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.756313 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfv9r\" (UniqueName: \"kubernetes.io/projected/e0c95901-6cf5-4cd4-9e0f-4c891aa1d591-kube-api-access-pfv9r\") pod \"llmisvc-controller-manager-869d5dcc5d-sdqcg\" (UID: \"e0c95901-6cf5-4cd4-9e0f-4c891aa1d591\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" Oct 31 00:39:38.756506 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.756338 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.756506 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.756361 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.757143 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.757109 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.758568 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.758522 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.759054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.759031 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.759155 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.759094 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.759284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.759261 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.759555 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.759538 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0c95901-6cf5-4cd4-9e0f-4c891aa1d591-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sdqcg\" (UID: \"e0c95901-6cf5-4cd4-9e0f-4c891aa1d591\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" Oct 31 00:39:38.767462 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.767436 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t54kt\" (UniqueName: \"kubernetes.io/projected/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-kube-api-access-t54kt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.769221 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.769194 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18daa545-6a52-4ba9-af4b-b60b4ac1f98c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnw2v\" (UID: \"18daa545-6a52-4ba9-af4b-b60b4ac1f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:38.769803 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.769783 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfv9r\" (UniqueName: \"kubernetes.io/projected/e0c95901-6cf5-4cd4-9e0f-4c891aa1d591-kube-api-access-pfv9r\") pod \"llmisvc-controller-manager-869d5dcc5d-sdqcg\" (UID: \"e0c95901-6cf5-4cd4-9e0f-4c891aa1d591\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" Oct 31 00:39:38.828451 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.828355 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:38.911387 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.911347 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" Oct 31 00:39:38.926948 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.926913 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:38.927898 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:38.927873 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:39.162931 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:39.162852 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:39.191973 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:39.191939 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg"] Oct 31 00:39:39.194413 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:39:39.194379 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode0c95901_6cf5_4cd4_9e0f_4c891aa1d591.slice/crio-cdc8ceebf3c1e181bbfb10c942305d3ffb02b7458b513b35e56e630b8a9741c3 WatchSource:0}: Error finding container cdc8ceebf3c1e181bbfb10c942305d3ffb02b7458b513b35e56e630b8a9741c3: Status 404 returned error can't find the container with id cdc8ceebf3c1e181bbfb10c942305d3ffb02b7458b513b35e56e630b8a9741c3 Oct 31 00:39:39.220989 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:39.220963 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v"] Oct 31 00:39:39.223877 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:39:39.223843 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18daa545_6a52_4ba9_af4b_b60b4ac1f98c.slice/crio-d560700f41433c4aed32c3c4aa7315779d32bf1828b69c3cd753c3759a3a8a8a WatchSource:0}: Error finding container d560700f41433c4aed32c3c4aa7315779d32bf1828b69c3cd753c3759a3a8a8a: Status 404 returned error can't find the container with id d560700f41433c4aed32c3c4aa7315779d32bf1828b69c3cd753c3759a3a8a8a Oct 31 00:39:39.225515 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:39.225484 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:39:39.225579 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:39.225549 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:39:39.508281 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:39.508246 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" event={"ID":"e0c95901-6cf5-4cd4-9e0f-4c891aa1d591","Type":"ContainerStarted","Data":"cdc8ceebf3c1e181bbfb10c942305d3ffb02b7458b513b35e56e630b8a9741c3"} Oct 31 00:39:39.509860 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:39.509823 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" event={"ID":"18daa545-6a52-4ba9-af4b-b60b4ac1f98c","Type":"ContainerStarted","Data":"4bc02f06f181374119414f83f5b43bde8b579644b974c96c565e5135e290e29b"} Oct 31 00:39:39.509984 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:39.509868 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" event={"ID":"18daa545-6a52-4ba9-af4b-b60b4ac1f98c","Type":"ContainerStarted","Data":"d560700f41433c4aed32c3c4aa7315779d32bf1828b69c3cd753c3759a3a8a8a"} Oct 31 00:39:39.509984 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:39.509912 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:39.534486 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:39.534435 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" podStartSLOduration=1.534420056 podStartE2EDuration="1.534420056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:39.532528773 +0000 UTC m=+375.561709175" watchObservedRunningTime="2026-10-31 00:39:39.534420056 +0000 UTC m=+375.563600445" Oct 31 00:39:39.679736 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:39.679693 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:39.780680 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:39.780578 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:39.860055 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:39.860020 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:39.925674 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:39.925612 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:40.515625 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.515597 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnw2v" Oct 31 00:39:40.568064 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.568024 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn"] Oct 31 00:39:40.568242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.568220 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" podUID="f71c13e9-da5c-44e2-9bb5-e40418b1544d" containerName="discovery" containerID="cri-o://6001200c2de4903b65625863aba623fe19b20af3074b43043cda6d6e432c50d6" gracePeriod=30 Oct 31 00:39:40.835863 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.835834 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:39:40.880993 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.880955 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-csr-ca-configmap\") pod \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " Oct 31 00:39:40.881175 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.881030 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-token\" (UniqueName: \"kubernetes.io/projected/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-token\") pod \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " Oct 31 00:39:40.881175 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.881074 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-kubeconfig\") pod \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " Oct 31 00:39:40.881175 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.881103 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-csr-dns-cert\") pod \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " Oct 31 00:39:40.881349 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.881264 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f71c13e9-da5c-44e2-9bb5-e40418b1544d-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71c13e9-da5c-44e2-9bb5-e40418b1544d-local-certs\") pod \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " Oct 31 00:39:40.881349 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.881333 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f71c13e9-da5c-44e2-9bb5-e40418b1544d-kube-api-access-j4zxz\" (UniqueName: \"kubernetes.io/projected/f71c13e9-da5c-44e2-9bb5-e40418b1544d-kube-api-access-j4zxz\") pod \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " Oct 31 00:39:40.881447 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.881387 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-cacerts\" (UniqueName: \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-cacerts\") pod \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\" (UID: \"f71c13e9-da5c-44e2-9bb5-e40418b1544d\") " Oct 31 00:39:40.881447 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.881413 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-csr-ca-configmap" pod "f71c13e9-da5c-44e2-9bb5-e40418b1544d" (UID: "f71c13e9-da5c-44e2-9bb5-e40418b1544d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:40.881711 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.881687 4519 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-csr-ca-configmap\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:40.884187 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.884158 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-kubeconfig" pod "f71c13e9-da5c-44e2-9bb5-e40418b1544d" (UID: "f71c13e9-da5c-44e2-9bb5-e40418b1544d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:40.884325 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.884230 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71c13e9-da5c-44e2-9bb5-e40418b1544d-local-certs" pod "f71c13e9-da5c-44e2-9bb5-e40418b1544d" (UID: "f71c13e9-da5c-44e2-9bb5-e40418b1544d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:39:40.884449 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.884428 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-csr-dns-cert" pod "f71c13e9-da5c-44e2-9bb5-e40418b1544d" (UID: "f71c13e9-da5c-44e2-9bb5-e40418b1544d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:40.884541 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.884502 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-token" pod "f71c13e9-da5c-44e2-9bb5-e40418b1544d" (UID: "f71c13e9-da5c-44e2-9bb5-e40418b1544d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:40.884825 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.884803 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-cacerts" pod "f71c13e9-da5c-44e2-9bb5-e40418b1544d" (UID: "f71c13e9-da5c-44e2-9bb5-e40418b1544d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:40.886116 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.886086 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71c13e9-da5c-44e2-9bb5-e40418b1544d-kube-api-access-j4zxz" pod "f71c13e9-da5c-44e2-9bb5-e40418b1544d" (UID: "f71c13e9-da5c-44e2-9bb5-e40418b1544d"). InnerVolumeSpecName "kube-api-access-j4zxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:40.982335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.982298 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:40.982335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.982337 4519 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-kubeconfig\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:40.982547 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.982351 4519 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-istio-csr-dns-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:40.982547 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.982363 4519 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71c13e9-da5c-44e2-9bb5-e40418b1544d-local-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:40.982547 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.982372 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4zxz\" (UniqueName: \"kubernetes.io/projected/f71c13e9-da5c-44e2-9bb5-e40418b1544d-kube-api-access-j4zxz\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:40.982547 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:40.982383 4519 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f71c13e9-da5c-44e2-9bb5-e40418b1544d-cacerts\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:41.519818 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:41.519702 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" event={"ID":"e0c95901-6cf5-4cd4-9e0f-4c891aa1d591","Type":"ContainerStarted","Data":"42a8f4d2ef39da71304b3fbc1682250380aa1008f7d66cc4fd5f2d21b4dc9160"} Oct 31 00:39:41.520239 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:41.519880 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" Oct 31 00:39:41.520935 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:41.520910 4519 generic.go:356] "Generic (PLEG): container finished" podID="f71c13e9-da5c-44e2-9bb5-e40418b1544d" containerID="6001200c2de4903b65625863aba623fe19b20af3074b43043cda6d6e432c50d6" exitCode=0 Oct 31 00:39:41.521051 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:41.520948 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" Oct 31 00:39:41.521051 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:41.520988 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" event={"ID":"f71c13e9-da5c-44e2-9bb5-e40418b1544d","Type":"ContainerDied","Data":"6001200c2de4903b65625863aba623fe19b20af3074b43043cda6d6e432c50d6"} Oct 31 00:39:41.521051 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:41.521028 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn" event={"ID":"f71c13e9-da5c-44e2-9bb5-e40418b1544d","Type":"ContainerDied","Data":"0468c3ce398114ccbc5d8f6e06c72bf7bc5673efd30542ac7d83841ca2d8588e"} Oct 31 00:39:41.521051 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:41.521049 4519 scope.go:122] "RemoveContainer" containerID="6001200c2de4903b65625863aba623fe19b20af3074b43043cda6d6e432c50d6" Oct 31 00:39:41.535769 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:41.535741 4519 scope.go:122] "RemoveContainer" containerID="6001200c2de4903b65625863aba623fe19b20af3074b43043cda6d6e432c50d6" Oct 31 00:39:41.536128 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:39:41.536100 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6001200c2de4903b65625863aba623fe19b20af3074b43043cda6d6e432c50d6\": container with ID starting with 6001200c2de4903b65625863aba623fe19b20af3074b43043cda6d6e432c50d6 not found: ID does not exist" containerID="6001200c2de4903b65625863aba623fe19b20af3074b43043cda6d6e432c50d6" Oct 31 00:39:41.536196 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:41.536141 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6001200c2de4903b65625863aba623fe19b20af3074b43043cda6d6e432c50d6"} err="failed to get container status \"6001200c2de4903b65625863aba623fe19b20af3074b43043cda6d6e432c50d6\": rpc error: code = NotFound desc = could not find container \"6001200c2de4903b65625863aba623fe19b20af3074b43043cda6d6e432c50d6\": container with ID starting with 6001200c2de4903b65625863aba623fe19b20af3074b43043cda6d6e432c50d6 not found: ID does not exist" Oct 31 00:39:41.538573 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:41.538525 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" podStartSLOduration=1.496755495 podStartE2EDuration="3.538512916s" totalImagesPullingTime="2.041757421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:41.537312678 +0000 UTC m=+377.566493070" watchObservedRunningTime="2026-10-31 00:39:41.538512916 +0000 UTC m=+377.567693306" Oct 31 00:39:41.553386 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:41.553353 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn"] Oct 31 00:39:41.557087 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:41.557061 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-dtdbn"] Oct 31 00:39:42.101339 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:42.101299 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f71c13e9-da5c-44e2-9bb5-e40418b1544d" path="/var/lib/kubelet/pods/f71c13e9-da5c-44e2-9bb5-e40418b1544d/volumes" Oct 31 00:39:43.857117 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:43.857083 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:43.958239 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:43.958208 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:46.249131 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.249096 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls"] Oct 31 00:39:46.249621 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.249560 4519 cpu_manager.go:414] "containerMap: removing container" podUID="f71c13e9-da5c-44e2-9bb5-e40418b1544d" containerName="discovery" Oct 31 00:39:46.249621 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.249575 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f71c13e9-da5c-44e2-9bb5-e40418b1544d" containerName="discovery" Oct 31 00:39:46.249762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.249656 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f71c13e9-da5c-44e2-9bb5-e40418b1544d" containerName="discovery" Oct 31 00:39:46.254906 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.254879 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" Oct 31 00:39:46.264610 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.262939 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 00:39:46.264610 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.262959 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 00:39:46.264610 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.263067 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 00:39:46.264610 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.263274 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-9gm5t\"" Oct 31 00:39:46.266189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.266159 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls"] Oct 31 00:39:46.294836 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.294799 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:46.326267 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.326223 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c0bb009c-f110-4275-8030-46207de5b3e2-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hkvls\" (UID: \"c0bb009c-f110-4275-8030-46207de5b3e2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" Oct 31 00:39:46.326471 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.326280 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf4z4\" (UniqueName: \"kubernetes.io/projected/c0bb009c-f110-4275-8030-46207de5b3e2-kube-api-access-zf4z4\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hkvls\" (UID: \"c0bb009c-f110-4275-8030-46207de5b3e2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" Oct 31 00:39:46.326471 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.326343 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0bb009c-f110-4275-8030-46207de5b3e2-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hkvls\" (UID: \"c0bb009c-f110-4275-8030-46207de5b3e2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" Oct 31 00:39:46.326471 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.326424 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0bb009c-f110-4275-8030-46207de5b3e2-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hkvls\" (UID: \"c0bb009c-f110-4275-8030-46207de5b3e2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" Oct 31 00:39:46.368163 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.368112 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:46.427017 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.426979 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0bb009c-f110-4275-8030-46207de5b3e2-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hkvls\" (UID: \"c0bb009c-f110-4275-8030-46207de5b3e2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" Oct 31 00:39:46.427017 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.427027 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c0bb009c-f110-4275-8030-46207de5b3e2-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hkvls\" (UID: \"c0bb009c-f110-4275-8030-46207de5b3e2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" Oct 31 00:39:46.427301 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.427056 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf4z4\" (UniqueName: \"kubernetes.io/projected/c0bb009c-f110-4275-8030-46207de5b3e2-kube-api-access-zf4z4\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hkvls\" (UID: \"c0bb009c-f110-4275-8030-46207de5b3e2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" Oct 31 00:39:46.427301 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.427076 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0bb009c-f110-4275-8030-46207de5b3e2-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hkvls\" (UID: \"c0bb009c-f110-4275-8030-46207de5b3e2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" Oct 31 00:39:46.427781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.427759 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0bb009c-f110-4275-8030-46207de5b3e2-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hkvls\" (UID: \"c0bb009c-f110-4275-8030-46207de5b3e2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" Oct 31 00:39:46.427975 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.427954 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c0bb009c-f110-4275-8030-46207de5b3e2-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hkvls\" (UID: \"c0bb009c-f110-4275-8030-46207de5b3e2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" Oct 31 00:39:46.429525 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.429504 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0bb009c-f110-4275-8030-46207de5b3e2-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hkvls\" (UID: \"c0bb009c-f110-4275-8030-46207de5b3e2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" Oct 31 00:39:46.435727 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.435706 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf4z4\" (UniqueName: \"kubernetes.io/projected/c0bb009c-f110-4275-8030-46207de5b3e2-kube-api-access-zf4z4\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hkvls\" (UID: \"c0bb009c-f110-4275-8030-46207de5b3e2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" Oct 31 00:39:46.465105 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.465071 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:46.570781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.570683 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" Oct 31 00:39:46.843095 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:46.843067 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls"] Oct 31 00:39:46.845268 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:39:46.845239 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0bb009c_f110_4275_8030_46207de5b3e2.slice/crio-9437bc750c2f3a72e981ab188f3b30bb1036f37ddafbe67c3add315780dc82eb WatchSource:0}: Error finding container 9437bc750c2f3a72e981ab188f3b30bb1036f37ddafbe67c3add315780dc82eb: Status 404 returned error can't find the container with id 9437bc750c2f3a72e981ab188f3b30bb1036f37ddafbe67c3add315780dc82eb Oct 31 00:39:47.541613 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:39:47.541574 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:39:47.546850 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:47.546822 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" event={"ID":"c0bb009c-f110-4275-8030-46207de5b3e2","Type":"ContainerStarted","Data":"9437bc750c2f3a72e981ab188f3b30bb1036f37ddafbe67c3add315780dc82eb"} Oct 31 00:39:57.574714 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:57.574138 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:58.608399 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:58.608360 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" event={"ID":"c0bb009c-f110-4275-8030-46207de5b3e2","Type":"ContainerStarted","Data":"25f1f5efe03cf4e815c2e4380dde9c0330b0e0bdd07cf2c40e7c578aa1171608"} Oct 31 00:39:58.608855 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:58.608462 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" Oct 31 00:39:58.626322 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:39:58.626266 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" podStartSLOduration=1.260093565 podStartE2EDuration="12.626251274s" totalImagesPullingTime="11.366157709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:58.62494419 +0000 UTC m=+394.654124581" watchObservedRunningTime="2026-10-31 00:39:58.626251274 +0000 UTC m=+394.655431663" Oct 31 00:40:09.619368 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:09.619289 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hkvls" Oct 31 00:40:10.031067 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:10.031033 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:12.528871 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:12.528836 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" Oct 31 00:40:14.802702 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:14.802663 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:14.900982 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:14.900945 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:16.044939 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:16.044885 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:16.146824 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:16.146786 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:16.215267 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:16.215238 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:17.541499 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:40:17.541461 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:40:26.142839 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:26.142808 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:26.247053 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:26.247022 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:26.458235 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:26.458152 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2xs7f"] Oct 31 00:40:26.466859 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:26.466828 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2xs7f" Oct 31 00:40:26.468188 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:26.468159 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2xs7f"] Oct 31 00:40:26.543134 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:26.543104 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2xs7f"] Oct 31 00:40:26.543309 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:40:26.543296 4519 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5cwbz], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-2xs7f" podUID="6f50fcd3-9e1c-48e0-b1ab-2908f299b90e" Oct 31 00:40:26.574458 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:26.574429 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cwbz\" (UniqueName: \"kubernetes.io/projected/6f50fcd3-9e1c-48e0-b1ab-2908f299b90e-kube-api-access-5cwbz\") pod \"authorino-6f6944bcdd-2xs7f\" (UID: \"6f50fcd3-9e1c-48e0-b1ab-2908f299b90e\") " pod="kuadrant-system/authorino-6f6944bcdd-2xs7f" Oct 31 00:40:26.674926 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:26.674893 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cwbz\" (UniqueName: \"kubernetes.io/projected/6f50fcd3-9e1c-48e0-b1ab-2908f299b90e-kube-api-access-5cwbz\") pod \"authorino-6f6944bcdd-2xs7f\" (UID: \"6f50fcd3-9e1c-48e0-b1ab-2908f299b90e\") " pod="kuadrant-system/authorino-6f6944bcdd-2xs7f" Oct 31 00:40:26.683245 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:26.683219 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cwbz\" (UniqueName: \"kubernetes.io/projected/6f50fcd3-9e1c-48e0-b1ab-2908f299b90e-kube-api-access-5cwbz\") pod \"authorino-6f6944bcdd-2xs7f\" (UID: \"6f50fcd3-9e1c-48e0-b1ab-2908f299b90e\") " pod="kuadrant-system/authorino-6f6944bcdd-2xs7f" Oct 31 00:40:26.725990 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:26.725917 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2xs7f" Oct 31 00:40:26.732798 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:26.732774 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2xs7f" Oct 31 00:40:26.876977 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:26.876942 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f50fcd3-9e1c-48e0-b1ab-2908f299b90e-kube-api-access-5cwbz\" (UniqueName: \"kubernetes.io/projected/6f50fcd3-9e1c-48e0-b1ab-2908f299b90e-kube-api-access-5cwbz\") pod \"6f50fcd3-9e1c-48e0-b1ab-2908f299b90e\" (UID: \"6f50fcd3-9e1c-48e0-b1ab-2908f299b90e\") " Oct 31 00:40:26.879202 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:26.879173 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f50fcd3-9e1c-48e0-b1ab-2908f299b90e-kube-api-access-5cwbz" pod "6f50fcd3-9e1c-48e0-b1ab-2908f299b90e" (UID: "6f50fcd3-9e1c-48e0-b1ab-2908f299b90e"). InnerVolumeSpecName "kube-api-access-5cwbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:26.978499 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:26.978407 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cwbz\" (UniqueName: \"kubernetes.io/projected/6f50fcd3-9e1c-48e0-b1ab-2908f299b90e-kube-api-access-5cwbz\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:27.730323 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:27.730284 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2xs7f" Oct 31 00:40:27.763128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:27.763097 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2xs7f"] Oct 31 00:40:27.764805 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:27.764766 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2xs7f"] Oct 31 00:40:28.102264 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:28.102184 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f50fcd3-9e1c-48e0-b1ab-2908f299b90e" path="/var/lib/kubelet/pods/6f50fcd3-9e1c-48e0-b1ab-2908f299b90e/volumes" Oct 31 00:40:29.272266 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:29.272233 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-msz8z"] Oct 31 00:40:29.272848 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:29.272421 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-msz8z" podUID="75f2bea7-a08e-4c6f-a405-b64c48fb85cc" containerName="authorino" containerID="cri-o://0f993f5cda9ac1f866d5e2fa363c431d41a6708ba0db878c7523c68317ccf8ff" gracePeriod=30 Oct 31 00:40:29.740047 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:29.740012 4519 generic.go:356] "Generic (PLEG): container finished" podID="75f2bea7-a08e-4c6f-a405-b64c48fb85cc" containerID="0f993f5cda9ac1f866d5e2fa363c431d41a6708ba0db878c7523c68317ccf8ff" exitCode=0 Oct 31 00:40:29.740208 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:29.740050 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-msz8z" event={"ID":"75f2bea7-a08e-4c6f-a405-b64c48fb85cc","Type":"ContainerDied","Data":"0f993f5cda9ac1f866d5e2fa363c431d41a6708ba0db878c7523c68317ccf8ff"} Oct 31 00:40:30.068148 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:30.068113 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-msz8z" Oct 31 00:40:30.103429 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:30.103394 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f2bea7-a08e-4c6f-a405-b64c48fb85cc-kube-api-access-2xq6m\" (UniqueName: \"kubernetes.io/projected/75f2bea7-a08e-4c6f-a405-b64c48fb85cc-kube-api-access-2xq6m\") pod \"75f2bea7-a08e-4c6f-a405-b64c48fb85cc\" (UID: \"75f2bea7-a08e-4c6f-a405-b64c48fb85cc\") " Oct 31 00:40:30.105532 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:30.105510 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f2bea7-a08e-4c6f-a405-b64c48fb85cc-kube-api-access-2xq6m" pod "75f2bea7-a08e-4c6f-a405-b64c48fb85cc" (UID: "75f2bea7-a08e-4c6f-a405-b64c48fb85cc"). InnerVolumeSpecName "kube-api-access-2xq6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:30.204242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:30.204207 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xq6m\" (UniqueName: \"kubernetes.io/projected/75f2bea7-a08e-4c6f-a405-b64c48fb85cc-kube-api-access-2xq6m\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:30.744605 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:30.744576 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-msz8z" Oct 31 00:40:30.744605 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:30.744585 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-msz8z" event={"ID":"75f2bea7-a08e-4c6f-a405-b64c48fb85cc","Type":"ContainerDied","Data":"78e5cdc0a46e3b98ee6a383f4d28c2cd93499e3d4be9c74d6154d69f25d95521"} Oct 31 00:40:30.745172 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:30.744630 4519 scope.go:122] "RemoveContainer" containerID="0f993f5cda9ac1f866d5e2fa363c431d41a6708ba0db878c7523c68317ccf8ff" Oct 31 00:40:30.766307 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:30.766278 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-msz8z"] Oct 31 00:40:30.770103 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:30.770079 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-msz8z"] Oct 31 00:40:32.101039 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:32.100997 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f2bea7-a08e-4c6f-a405-b64c48fb85cc" path="/var/lib/kubelet/pods/75f2bea7-a08e-4c6f-a405-b64c48fb85cc/volumes" Oct 31 00:40:43.122767 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:43.122723 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:43.223311 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:43.223281 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:47.540909 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:40:47.540861 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:40:50.378231 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:50.378195 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-hzz82"] Oct 31 00:40:50.378631 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:50.378501 4519 cpu_manager.go:414] "containerMap: removing container" podUID="75f2bea7-a08e-4c6f-a405-b64c48fb85cc" containerName="authorino" Oct 31 00:40:50.378631 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:50.378513 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f2bea7-a08e-4c6f-a405-b64c48fb85cc" containerName="authorino" Oct 31 00:40:50.378631 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:50.378559 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f2bea7-a08e-4c6f-a405-b64c48fb85cc" containerName="authorino" Oct 31 00:40:50.381609 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:50.381589 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hzz82" Oct 31 00:40:50.393259 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:50.393231 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-hzz82"] Oct 31 00:40:50.441245 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:50.441205 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gktpz\" (UniqueName: \"kubernetes.io/projected/3147d13d-fa94-4a4c-a468-b7c9e275ef43-kube-api-access-gktpz\") pod \"llmisvc-controller-manager-6466c554d8-hzz82\" (UID: \"3147d13d-fa94-4a4c-a468-b7c9e275ef43\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hzz82" Oct 31 00:40:50.441421 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:50.441279 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3147d13d-fa94-4a4c-a468-b7c9e275ef43-cert\") pod \"llmisvc-controller-manager-6466c554d8-hzz82\" (UID: \"3147d13d-fa94-4a4c-a468-b7c9e275ef43\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hzz82" Oct 31 00:40:50.541762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:50.541729 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3147d13d-fa94-4a4c-a468-b7c9e275ef43-cert\") pod \"llmisvc-controller-manager-6466c554d8-hzz82\" (UID: \"3147d13d-fa94-4a4c-a468-b7c9e275ef43\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hzz82" Oct 31 00:40:50.541905 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:50.541799 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gktpz\" (UniqueName: \"kubernetes.io/projected/3147d13d-fa94-4a4c-a468-b7c9e275ef43-kube-api-access-gktpz\") pod \"llmisvc-controller-manager-6466c554d8-hzz82\" (UID: \"3147d13d-fa94-4a4c-a468-b7c9e275ef43\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hzz82" Oct 31 00:40:50.544380 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:50.544357 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3147d13d-fa94-4a4c-a468-b7c9e275ef43-cert\") pod \"llmisvc-controller-manager-6466c554d8-hzz82\" (UID: \"3147d13d-fa94-4a4c-a468-b7c9e275ef43\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hzz82" Oct 31 00:40:50.551285 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:50.551261 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gktpz\" (UniqueName: \"kubernetes.io/projected/3147d13d-fa94-4a4c-a468-b7c9e275ef43-kube-api-access-gktpz\") pod \"llmisvc-controller-manager-6466c554d8-hzz82\" (UID: \"3147d13d-fa94-4a4c-a468-b7c9e275ef43\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hzz82" Oct 31 00:40:50.693999 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:50.693918 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hzz82" Oct 31 00:40:50.967676 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:50.967622 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-hzz82"] Oct 31 00:40:50.969604 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:40:50.969566 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3147d13d_fa94_4a4c_a468_b7c9e275ef43.slice/crio-1051bef072aba9643b6e9808752e355df663cbb694b8fd18e2a4fb72d60541df WatchSource:0}: Error finding container 1051bef072aba9643b6e9808752e355df663cbb694b8fd18e2a4fb72d60541df: Status 404 returned error can't find the container with id 1051bef072aba9643b6e9808752e355df663cbb694b8fd18e2a4fb72d60541df Oct 31 00:40:51.825645 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:51.825591 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hzz82" event={"ID":"3147d13d-fa94-4a4c-a468-b7c9e275ef43","Type":"ContainerStarted","Data":"e515071080d50eb875143379ff9722fbfd7890396c811763b7ff3c009b33b7e9"} Oct 31 00:40:51.826104 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:51.825688 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hzz82" event={"ID":"3147d13d-fa94-4a4c-a468-b7c9e275ef43","Type":"ContainerStarted","Data":"1051bef072aba9643b6e9808752e355df663cbb694b8fd18e2a4fb72d60541df"} Oct 31 00:40:51.826104 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:51.825712 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hzz82" Oct 31 00:40:51.846940 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:40:51.846871 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hzz82" podStartSLOduration=1.415545286 podStartE2EDuration="1.846858726s" totalImagesPullingTime="431.31344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:51.844370588 +0000 UTC m=+447.873551001" watchObservedRunningTime="2026-10-31 00:40:51.846858726 +0000 UTC m=+447.876039113" Oct 31 00:41:10.088687 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:10.088645 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:17.542088 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:41:17.542046 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:41:22.831359 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:22.831329 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hzz82" Oct 31 00:41:22.888315 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:22.888280 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg"] Oct 31 00:41:22.888537 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:22.888516 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" podUID="e0c95901-6cf5-4cd4-9e0f-4c891aa1d591" containerName="manager" containerID="cri-o://42a8f4d2ef39da71304b3fbc1682250380aa1008f7d66cc4fd5f2d21b4dc9160" gracePeriod=30 Oct 31 00:41:23.127348 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.127324 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" Oct 31 00:41:23.201084 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.201050 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c95901-6cf5-4cd4-9e0f-4c891aa1d591-cert\" (UniqueName: \"kubernetes.io/secret/e0c95901-6cf5-4cd4-9e0f-4c891aa1d591-cert\") pod \"e0c95901-6cf5-4cd4-9e0f-4c891aa1d591\" (UID: \"e0c95901-6cf5-4cd4-9e0f-4c891aa1d591\") " Oct 31 00:41:23.201254 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.201133 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c95901-6cf5-4cd4-9e0f-4c891aa1d591-kube-api-access-pfv9r\" (UniqueName: \"kubernetes.io/projected/e0c95901-6cf5-4cd4-9e0f-4c891aa1d591-kube-api-access-pfv9r\") pod \"e0c95901-6cf5-4cd4-9e0f-4c891aa1d591\" (UID: \"e0c95901-6cf5-4cd4-9e0f-4c891aa1d591\") " Oct 31 00:41:23.203384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.203353 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c95901-6cf5-4cd4-9e0f-4c891aa1d591-cert" pod "e0c95901-6cf5-4cd4-9e0f-4c891aa1d591" (UID: "e0c95901-6cf5-4cd4-9e0f-4c891aa1d591"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:41:23.203384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.203353 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c95901-6cf5-4cd4-9e0f-4c891aa1d591-kube-api-access-pfv9r" pod "e0c95901-6cf5-4cd4-9e0f-4c891aa1d591" (UID: "e0c95901-6cf5-4cd4-9e0f-4c891aa1d591"). InnerVolumeSpecName "kube-api-access-pfv9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:23.302229 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.302199 4519 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0c95901-6cf5-4cd4-9e0f-4c891aa1d591-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:23.302229 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.302227 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfv9r\" (UniqueName: \"kubernetes.io/projected/e0c95901-6cf5-4cd4-9e0f-4c891aa1d591-kube-api-access-pfv9r\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:23.729587 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.729557 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:23.828136 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.828103 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:23.948968 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.948934 4519 generic.go:356] "Generic (PLEG): container finished" podID="e0c95901-6cf5-4cd4-9e0f-4c891aa1d591" containerID="42a8f4d2ef39da71304b3fbc1682250380aa1008f7d66cc4fd5f2d21b4dc9160" exitCode=0 Oct 31 00:41:23.949454 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.948977 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" Oct 31 00:41:23.949454 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.948982 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" event={"ID":"e0c95901-6cf5-4cd4-9e0f-4c891aa1d591","Type":"ContainerDied","Data":"42a8f4d2ef39da71304b3fbc1682250380aa1008f7d66cc4fd5f2d21b4dc9160"} Oct 31 00:41:23.949454 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.949034 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg" event={"ID":"e0c95901-6cf5-4cd4-9e0f-4c891aa1d591","Type":"ContainerDied","Data":"cdc8ceebf3c1e181bbfb10c942305d3ffb02b7458b513b35e56e630b8a9741c3"} Oct 31 00:41:23.949454 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.949067 4519 scope.go:122] "RemoveContainer" containerID="42a8f4d2ef39da71304b3fbc1682250380aa1008f7d66cc4fd5f2d21b4dc9160" Oct 31 00:41:23.957255 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.957236 4519 scope.go:122] "RemoveContainer" containerID="42a8f4d2ef39da71304b3fbc1682250380aa1008f7d66cc4fd5f2d21b4dc9160" Oct 31 00:41:23.957526 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:41:23.957507 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a8f4d2ef39da71304b3fbc1682250380aa1008f7d66cc4fd5f2d21b4dc9160\": container with ID starting with 42a8f4d2ef39da71304b3fbc1682250380aa1008f7d66cc4fd5f2d21b4dc9160 not found: ID does not exist" containerID="42a8f4d2ef39da71304b3fbc1682250380aa1008f7d66cc4fd5f2d21b4dc9160" Oct 31 00:41:23.957570 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.957536 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a8f4d2ef39da71304b3fbc1682250380aa1008f7d66cc4fd5f2d21b4dc9160"} err="failed to get container status \"42a8f4d2ef39da71304b3fbc1682250380aa1008f7d66cc4fd5f2d21b4dc9160\": rpc error: code = NotFound desc = could not find container \"42a8f4d2ef39da71304b3fbc1682250380aa1008f7d66cc4fd5f2d21b4dc9160\": container with ID starting with 42a8f4d2ef39da71304b3fbc1682250380aa1008f7d66cc4fd5f2d21b4dc9160 not found: ID does not exist" Oct 31 00:41:23.974069 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.974043 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg"] Oct 31 00:41:23.978680 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:23.978653 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sdqcg"] Oct 31 00:41:24.099962 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:24.099933 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c95901-6cf5-4cd4-9e0f-4c891aa1d591" path="/var/lib/kubelet/pods/e0c95901-6cf5-4cd4-9e0f-4c891aa1d591/volumes" Oct 31 00:41:25.193740 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:25.193707 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:45.281188 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:45.281096 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:46.415674 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:46.415622 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:47.540865 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:41:47.540826 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:41:48.213536 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:48.213499 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:48.962129 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:48.962094 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:50.344437 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:50.344390 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:51.772204 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:51.772168 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:52.825764 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:52.825731 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:54.214668 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:54.214621 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:57.942496 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:57.942457 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m"] Oct 31 00:41:57.942998 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:57.942780 4519 cpu_manager.go:414] "containerMap: removing container" podUID="e0c95901-6cf5-4cd4-9e0f-4c891aa1d591" containerName="manager" Oct 31 00:41:57.942998 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:57.942792 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c95901-6cf5-4cd4-9e0f-4c891aa1d591" containerName="manager" Oct 31 00:41:57.942998 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:57.942858 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c95901-6cf5-4cd4-9e0f-4c891aa1d591" containerName="manager" Oct 31 00:41:57.945831 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:57.945810 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:57.968184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:57.968155 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m"] Oct 31 00:41:57.991563 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:57.991527 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:57.991729 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:57.991572 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:57.991729 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:57.991709 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:57.991862 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:57.991744 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:57.991862 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:57.991761 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:57.991862 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:57.991778 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:57.991862 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:57.991814 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:57.992026 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:57.991888 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:57.992026 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:57.991917 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9xpq\" (UniqueName: \"kubernetes.io/projected/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-kube-api-access-d9xpq\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.030142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.030108 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-drfvd\"" Oct 31 00:41:58.092474 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.092441 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.092762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.092482 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.092762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.092507 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.092762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.092538 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.092762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.092564 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.092762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.092585 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.092762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.092612 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.092762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.092709 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.093127 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.092796 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9xpq\" (UniqueName: \"kubernetes.io/projected/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-kube-api-access-d9xpq\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.093127 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.092970 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.093127 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.093046 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.093127 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.093083 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.093521 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.093504 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.093966 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.093941 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.095186 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.095157 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.095186 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.095165 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.101487 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.101465 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.101744 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.101724 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9xpq\" (UniqueName: \"kubernetes.io/projected/a11bc6fe-fa8b-425e-b6df-007d3fc6aa07-kube-api-access-d9xpq\") pod \"maas-default-gateway-openshift-default-6f54779779-8wb6m\" (UID: \"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.260693 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.260660 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:41:58.539606 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:58.539579 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m"] Oct 31 00:41:58.541365 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:41:58.541334 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11bc6fe_fa8b_425e_b6df_007d3fc6aa07.slice/crio-c6ad17ccd1945766da924ec4cce3aa21683a60ce5cad6c9203a71a0f98955f8a WatchSource:0}: Error finding container c6ad17ccd1945766da924ec4cce3aa21683a60ce5cad6c9203a71a0f98955f8a: Status 404 returned error can't find the container with id c6ad17ccd1945766da924ec4cce3aa21683a60ce5cad6c9203a71a0f98955f8a Oct 31 00:41:59.085175 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:41:59.085137 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" event={"ID":"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07","Type":"ContainerStarted","Data":"c6ad17ccd1945766da924ec4cce3aa21683a60ce5cad6c9203a71a0f98955f8a"} Oct 31 00:42:01.408068 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:01.408031 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:42:01.408369 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:01.408121 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:42:01.408369 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:01.408161 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:42:02.100340 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:02.100301 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" event={"ID":"a11bc6fe-fa8b-425e-b6df-007d3fc6aa07","Type":"ContainerStarted","Data":"ef6c0c571e7543856ac109a4104d799e2791f4babc6d608c5b2b5276f2001be6"} Oct 31 00:42:02.123994 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:02.123922 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" podStartSLOduration=2.260083071 podStartE2EDuration="5.123904046s" totalImagesPullingTime="2.863820975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:02.121459503 +0000 UTC m=+518.150639894" watchObservedRunningTime="2026-10-31 00:42:02.123904046 +0000 UTC m=+518.153084437" Oct 31 00:42:02.261059 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:02.261017 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:42:02.262391 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:02.262361 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8wb6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 31 00:42:02.262530 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:02.262426 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" podUID="a11bc6fe-fa8b-425e-b6df-007d3fc6aa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 31 00:42:03.261784 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:03.261746 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8wb6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 31 00:42:03.262206 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:03.261804 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" podUID="a11bc6fe-fa8b-425e-b6df-007d3fc6aa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 31 00:42:04.262053 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:04.262019 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8wb6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 31 00:42:04.262454 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:04.262070 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" podUID="a11bc6fe-fa8b-425e-b6df-007d3fc6aa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 31 00:42:05.261650 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:05.261599 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8wb6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 31 00:42:05.261837 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:05.261700 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" podUID="a11bc6fe-fa8b-425e-b6df-007d3fc6aa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 31 00:42:06.261724 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:06.261686 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8wb6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 31 00:42:06.262109 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:06.261760 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" podUID="a11bc6fe-fa8b-425e-b6df-007d3fc6aa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 31 00:42:07.262075 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:07.262041 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8wb6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 31 00:42:07.262481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:07.262099 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" podUID="a11bc6fe-fa8b-425e-b6df-007d3fc6aa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 31 00:42:08.261333 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:08.261294 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8wb6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 31 00:42:08.261541 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:08.261357 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" podUID="a11bc6fe-fa8b-425e-b6df-007d3fc6aa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 31 00:42:09.755782 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:09.755751 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:42:09.756592 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:09.756564 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:42:09.757157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:09.757135 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" Oct 31 00:42:10.122813 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:10.122730 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:12.515965 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:12.515931 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:12.624162 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:12.624132 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:12.957579 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:12.957486 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4"] Oct 31 00:42:12.961178 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:12.961152 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:12.971657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:12.971601 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4"] Oct 31 00:42:13.021976 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.021934 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a6c37d5-3025-470f-94c0-ee1502fc7a76-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.021976 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.021974 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j47l9\" (UniqueName: \"kubernetes.io/projected/1a6c37d5-3025-470f-94c0-ee1502fc7a76-kube-api-access-j47l9\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.022211 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.022015 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a6c37d5-3025-470f-94c0-ee1502fc7a76-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.022211 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.022159 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a6c37d5-3025-470f-94c0-ee1502fc7a76-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.022211 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.022203 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a6c37d5-3025-470f-94c0-ee1502fc7a76-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.022324 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.022285 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6c37d5-3025-470f-94c0-ee1502fc7a76-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.022362 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.022319 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a6c37d5-3025-470f-94c0-ee1502fc7a76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.022398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.022357 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a6c37d5-3025-470f-94c0-ee1502fc7a76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.022441 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.022391 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6c37d5-3025-470f-94c0-ee1502fc7a76-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.122764 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.122726 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6c37d5-3025-470f-94c0-ee1502fc7a76-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.122764 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.122768 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a6c37d5-3025-470f-94c0-ee1502fc7a76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.122996 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.122790 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a6c37d5-3025-470f-94c0-ee1502fc7a76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.122996 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.122905 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6c37d5-3025-470f-94c0-ee1502fc7a76-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.122996 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.122982 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a6c37d5-3025-470f-94c0-ee1502fc7a76-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.123153 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.123010 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j47l9\" (UniqueName: \"kubernetes.io/projected/1a6c37d5-3025-470f-94c0-ee1502fc7a76-kube-api-access-j47l9\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.123153 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.123044 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a6c37d5-3025-470f-94c0-ee1502fc7a76-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.123153 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.123087 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a6c37d5-3025-470f-94c0-ee1502fc7a76-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.123153 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.123117 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a6c37d5-3025-470f-94c0-ee1502fc7a76-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.123347 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.123261 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6c37d5-3025-470f-94c0-ee1502fc7a76-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.123347 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.123310 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a6c37d5-3025-470f-94c0-ee1502fc7a76-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.123443 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.123350 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a6c37d5-3025-470f-94c0-ee1502fc7a76-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.123561 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.123528 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a6c37d5-3025-470f-94c0-ee1502fc7a76-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.123761 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.123741 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a6c37d5-3025-470f-94c0-ee1502fc7a76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.125292 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.125270 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a6c37d5-3025-470f-94c0-ee1502fc7a76-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.125471 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.125454 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a6c37d5-3025-470f-94c0-ee1502fc7a76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.132489 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.132462 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a6c37d5-3025-470f-94c0-ee1502fc7a76-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.132766 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.132743 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j47l9\" (UniqueName: \"kubernetes.io/projected/1a6c37d5-3025-470f-94c0-ee1502fc7a76-kube-api-access-j47l9\") pod \"maas-default-gateway-openshift-default-6f54779779-ggdf4\" (UID: \"1a6c37d5-3025-470f-94c0-ee1502fc7a76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.275081 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.275037 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:13.549915 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.549890 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4"] Oct 31 00:42:13.552335 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:42:13.552293 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6c37d5_3025_470f_94c0_ee1502fc7a76.slice/crio-29b6ba129439c3788c44c6c2bb0c9f924e2b5d9157694260c3fcbb543c83d0bc WatchSource:0}: Error finding container 29b6ba129439c3788c44c6c2bb0c9f924e2b5d9157694260c3fcbb543c83d0bc: Status 404 returned error can't find the container with id 29b6ba129439c3788c44c6c2bb0c9f924e2b5d9157694260c3fcbb543c83d0bc Oct 31 00:42:13.556425 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.556395 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:42:13.556492 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.556460 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:42:13.556492 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:13.556488 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:42:14.168974 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:14.168934 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" event={"ID":"1a6c37d5-3025-470f-94c0-ee1502fc7a76","Type":"ContainerStarted","Data":"23dabda4461bbcd359bbd38fd3f47c8fe43cb65b41fe76f16b6bd5ea53133c02"} Oct 31 00:42:14.169163 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:14.169018 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" event={"ID":"1a6c37d5-3025-470f-94c0-ee1502fc7a76","Type":"ContainerStarted","Data":"29b6ba129439c3788c44c6c2bb0c9f924e2b5d9157694260c3fcbb543c83d0bc"} Oct 31 00:42:14.192073 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:14.192021 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" podStartSLOduration=2.1920072409999998 podStartE2EDuration="2.192007241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:14.190854647 +0000 UTC m=+530.220035074" watchObservedRunningTime="2026-10-31 00:42:14.192007241 +0000 UTC m=+530.221187631" Oct 31 00:42:14.275191 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:14.275166 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:14.276900 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:14.276870 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ggdf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:42:14.277054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:14.276947 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" podUID="1a6c37d5-3025-470f-94c0-ee1502fc7a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 00:42:15.275991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:15.275953 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ggdf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:42:15.276394 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:15.276009 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" podUID="1a6c37d5-3025-470f-94c0-ee1502fc7a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 00:42:16.275365 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:16.275326 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ggdf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:42:16.275625 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:16.275388 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" podUID="1a6c37d5-3025-470f-94c0-ee1502fc7a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 00:42:17.275958 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:17.275919 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ggdf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:42:17.276376 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:17.275976 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" podUID="1a6c37d5-3025-470f-94c0-ee1502fc7a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 00:42:17.558405 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:42:17.558322 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:42:18.276209 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:18.276173 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ggdf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:42:18.276616 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:18.276224 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" podUID="1a6c37d5-3025-470f-94c0-ee1502fc7a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 00:42:18.937335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:18.937297 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:19.048088 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:19.048055 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:19.275525 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:19.275486 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ggdf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:42:19.275728 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:19.275551 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" podUID="1a6c37d5-3025-470f-94c0-ee1502fc7a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 00:42:20.276280 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:20.276238 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ggdf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:42:20.276828 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:20.276326 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" podUID="1a6c37d5-3025-470f-94c0-ee1502fc7a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 00:42:22.275962 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:22.275918 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ggdf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:42:22.276441 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:22.275978 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" podUID="1a6c37d5-3025-470f-94c0-ee1502fc7a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:23.317848 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:23.317806 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ggdf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:42:23.318303 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:23.317881 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" podUID="1a6c37d5-3025-470f-94c0-ee1502fc7a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:24.052889 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:24.052857 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:24.053264 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:24.053231 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:24.054065 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:24.054044 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" Oct 31 00:42:44.656898 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:44.656866 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:45.204865 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:45.204832 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:46.915350 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:46.915316 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:47.021362 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:47.021329 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:47.540957 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:42:47.540854 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:42:48.403840 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:48.403800 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:48.502692 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:42:48.502625 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:10.171982 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:10.171896 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:11.707365 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:11.707328 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:11.794870 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:11.794837 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:12.294034 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:12.293999 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:12.402240 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:12.402192 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:12.501115 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:12.501079 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:12.605475 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:12.605379 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:12.709576 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:12.709541 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:12.799837 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:12.799804 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:12.882174 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:12.882075 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:12.954032 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:12.953996 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:13.336142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:13.336110 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:15.655429 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:15.655376 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:15.751807 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:15.751765 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:16.035773 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:16.035736 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:16.136559 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:16.136513 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:16.232901 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:16.232853 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:16.332783 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:16.332690 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:16.535302 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:16.535271 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:16.633701 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:16.633601 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:16.728307 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:16.728272 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:16.875173 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:16.875140 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:16.971674 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:16.971574 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:17.268223 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:17.268193 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:17.377248 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:17.377208 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:17.548964 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:43:17.548887 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:43:17.901089 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:17.901009 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:17.953773 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:17.953737 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k"] Oct 31 00:43:17.956934 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:17.956917 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" Oct 31 00:43:17.965865 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:17.965843 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-5c2a03\"" Oct 31 00:43:17.966017 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:17.965959 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-5c2a03-dockercfg-s9tk9\"" Oct 31 00:43:17.968843 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:17.968819 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k"] Oct 31 00:43:17.989121 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:17.989086 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:17.997840 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:17.997803 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptszr\" (UniqueName: \"kubernetes.io/projected/6b03043a-75b4-404a-800f-d71f33a04108-kube-api-access-ptszr\") pod \"payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k\" (UID: \"6b03043a-75b4-404a-800f-d71f33a04108\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" Oct 31 00:43:17.998002 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:17.997879 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b03043a-75b4-404a-800f-d71f33a04108-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k\" (UID: \"6b03043a-75b4-404a-800f-d71f33a04108\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" Oct 31 00:43:18.098597 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.098555 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b03043a-75b4-404a-800f-d71f33a04108-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k\" (UID: \"6b03043a-75b4-404a-800f-d71f33a04108\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" Oct 31 00:43:18.098816 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.098738 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptszr\" (UniqueName: \"kubernetes.io/projected/6b03043a-75b4-404a-800f-d71f33a04108-kube-api-access-ptszr\") pod \"payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k\" (UID: \"6b03043a-75b4-404a-800f-d71f33a04108\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" Oct 31 00:43:18.099491 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.099458 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b03043a-75b4-404a-800f-d71f33a04108-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k\" (UID: \"6b03043a-75b4-404a-800f-d71f33a04108\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" Oct 31 00:43:18.110303 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.110275 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptszr\" (UniqueName: \"kubernetes.io/projected/6b03043a-75b4-404a-800f-d71f33a04108-kube-api-access-ptszr\") pod \"payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k\" (UID: \"6b03043a-75b4-404a-800f-d71f33a04108\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" Oct 31 00:43:18.230089 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.230070 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:18.270851 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.270816 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" Oct 31 00:43:18.332094 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.332057 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:18.466700 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.466669 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:18.546726 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.546702 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k"] Oct 31 00:43:18.549447 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:43:18.549420 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b03043a_75b4_404a_800f_d71f33a04108.slice/crio-8869353d8c0f67f8e789008ead84f4c24c6e650fcc9c4fc1fef6d63d59034896 WatchSource:0}: Error finding container 8869353d8c0f67f8e789008ead84f4c24c6e650fcc9c4fc1fef6d63d59034896: Status 404 returned error can't find the container with id 8869353d8c0f67f8e789008ead84f4c24c6e650fcc9c4fc1fef6d63d59034896 Oct 31 00:43:18.676986 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.676951 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:18.727121 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.727080 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv"] Oct 31 00:43:18.732089 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.732069 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" Oct 31 00:43:18.741363 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.740577 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv"] Oct 31 00:43:18.741499 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.741475 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-44bc67\"" Oct 31 00:43:18.741619 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.741479 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-44bc67-dockercfg-mktgc\"" Oct 31 00:43:18.745995 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.745971 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp"] Oct 31 00:43:18.749042 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.749024 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" Oct 31 00:43:18.760345 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.760315 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp"] Oct 31 00:43:18.768195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.768171 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:18.804493 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.804415 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw764\" (UniqueName: \"kubernetes.io/projected/fd5bf7df-a3e9-4de3-ac84-e258676c6f92-kube-api-access-sw764\") pod \"payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp\" (UID: \"fd5bf7df-a3e9-4de3-ac84-e258676c6f92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" Oct 31 00:43:18.804493 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.804463 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw8lz\" (UniqueName: \"kubernetes.io/projected/7dd65b23-b72a-4eb4-971f-5f218554d0f2-kube-api-access-zw8lz\") pod \"payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv\" (UID: \"7dd65b23-b72a-4eb4-971f-5f218554d0f2\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" Oct 31 00:43:18.804743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.804611 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd65b23-b72a-4eb4-971f-5f218554d0f2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv\" (UID: \"7dd65b23-b72a-4eb4-971f-5f218554d0f2\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" Oct 31 00:43:18.804743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.804673 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5bf7df-a3e9-4de3-ac84-e258676c6f92-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp\" (UID: \"fd5bf7df-a3e9-4de3-ac84-e258676c6f92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" Oct 31 00:43:18.906475 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.906431 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw764\" (UniqueName: \"kubernetes.io/projected/fd5bf7df-a3e9-4de3-ac84-e258676c6f92-kube-api-access-sw764\") pod \"payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp\" (UID: \"fd5bf7df-a3e9-4de3-ac84-e258676c6f92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" Oct 31 00:43:18.907536 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.907502 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw8lz\" (UniqueName: \"kubernetes.io/projected/7dd65b23-b72a-4eb4-971f-5f218554d0f2-kube-api-access-zw8lz\") pod \"payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv\" (UID: \"7dd65b23-b72a-4eb4-971f-5f218554d0f2\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" Oct 31 00:43:18.907759 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.907582 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd65b23-b72a-4eb4-971f-5f218554d0f2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv\" (UID: \"7dd65b23-b72a-4eb4-971f-5f218554d0f2\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" Oct 31 00:43:18.907759 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.907611 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5bf7df-a3e9-4de3-ac84-e258676c6f92-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp\" (UID: \"fd5bf7df-a3e9-4de3-ac84-e258676c6f92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" Oct 31 00:43:18.908604 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.908558 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5bf7df-a3e9-4de3-ac84-e258676c6f92-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp\" (UID: \"fd5bf7df-a3e9-4de3-ac84-e258676c6f92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" Oct 31 00:43:18.908902 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.908882 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd65b23-b72a-4eb4-971f-5f218554d0f2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv\" (UID: \"7dd65b23-b72a-4eb4-971f-5f218554d0f2\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" Oct 31 00:43:18.916496 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.916469 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw8lz\" (UniqueName: \"kubernetes.io/projected/7dd65b23-b72a-4eb4-971f-5f218554d0f2-kube-api-access-zw8lz\") pod \"payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv\" (UID: \"7dd65b23-b72a-4eb4-971f-5f218554d0f2\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" Oct 31 00:43:18.916689 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:18.916665 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw764\" (UniqueName: \"kubernetes.io/projected/fd5bf7df-a3e9-4de3-ac84-e258676c6f92-kube-api-access-sw764\") pod \"payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp\" (UID: \"fd5bf7df-a3e9-4de3-ac84-e258676c6f92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" Oct 31 00:43:19.010487 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.010453 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:19.048764 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.048726 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" Oct 31 00:43:19.063879 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.063840 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" Oct 31 00:43:19.065021 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.064997 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm"] Oct 31 00:43:19.070197 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.070169 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" Oct 31 00:43:19.075505 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.075480 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-7f1496-dockercfg-7q65v\"" Oct 31 00:43:19.079281 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.079252 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm"] Oct 31 00:43:19.094258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.094223 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:19.110115 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.110082 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbj59\" (UniqueName: \"kubernetes.io/projected/a1a22d4f-183e-4d55-b8f5-71c4d30736bb-kube-api-access-gbj59\") pod \"payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm\" (UID: \"a1a22d4f-183e-4d55-b8f5-71c4d30736bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" Oct 31 00:43:19.110289 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.110149 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a22d4f-183e-4d55-b8f5-71c4d30736bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm\" (UID: \"a1a22d4f-183e-4d55-b8f5-71c4d30736bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" Oct 31 00:43:19.158132 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.158075 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-7f1496\"" Oct 31 00:43:19.219177 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.217212 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbj59\" (UniqueName: \"kubernetes.io/projected/a1a22d4f-183e-4d55-b8f5-71c4d30736bb-kube-api-access-gbj59\") pod \"payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm\" (UID: \"a1a22d4f-183e-4d55-b8f5-71c4d30736bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" Oct 31 00:43:19.219177 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.217304 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a22d4f-183e-4d55-b8f5-71c4d30736bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm\" (UID: \"a1a22d4f-183e-4d55-b8f5-71c4d30736bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" Oct 31 00:43:19.219177 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.218945 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a22d4f-183e-4d55-b8f5-71c4d30736bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm\" (UID: \"a1a22d4f-183e-4d55-b8f5-71c4d30736bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" Oct 31 00:43:19.242966 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.241585 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbj59\" (UniqueName: \"kubernetes.io/projected/a1a22d4f-183e-4d55-b8f5-71c4d30736bb-kube-api-access-gbj59\") pod \"payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm\" (UID: \"a1a22d4f-183e-4d55-b8f5-71c4d30736bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" Oct 31 00:43:19.244356 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.244174 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:19.349933 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.349798 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:19.376236 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.376201 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv"] Oct 31 00:43:19.378864 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:43:19.378833 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd65b23_b72a_4eb4_971f_5f218554d0f2.slice/crio-5ac8d277cebb97baafab1893856d0343e642ae92aaae1b20b49e0178f107bb93 WatchSource:0}: Error finding container 5ac8d277cebb97baafab1893856d0343e642ae92aaae1b20b49e0178f107bb93: Status 404 returned error can't find the container with id 5ac8d277cebb97baafab1893856d0343e642ae92aaae1b20b49e0178f107bb93 Oct 31 00:43:19.392087 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.392023 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp"] Oct 31 00:43:19.394540 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.394520 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" Oct 31 00:43:19.395014 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:43:19.394987 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5bf7df_a3e9_4de3_ac84_e258676c6f92.slice/crio-cb971269ad8e8c3cd1d588ce61b6c2345e5a93445701cd622bddc1502d87646d WatchSource:0}: Error finding container cb971269ad8e8c3cd1d588ce61b6c2345e5a93445701cd622bddc1502d87646d: Status 404 returned error can't find the container with id cb971269ad8e8c3cd1d588ce61b6c2345e5a93445701cd622bddc1502d87646d Oct 31 00:43:19.435458 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.435384 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" event={"ID":"fd5bf7df-a3e9-4de3-ac84-e258676c6f92","Type":"ContainerStarted","Data":"cb971269ad8e8c3cd1d588ce61b6c2345e5a93445701cd622bddc1502d87646d"} Oct 31 00:43:19.438252 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.438219 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" event={"ID":"6b03043a-75b4-404a-800f-d71f33a04108","Type":"ContainerStarted","Data":"8869353d8c0f67f8e789008ead84f4c24c6e650fcc9c4fc1fef6d63d59034896"} Oct 31 00:43:19.439561 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.439535 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" event={"ID":"7dd65b23-b72a-4eb4-971f-5f218554d0f2","Type":"ContainerStarted","Data":"5ac8d277cebb97baafab1893856d0343e642ae92aaae1b20b49e0178f107bb93"} Oct 31 00:43:19.479550 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.479510 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:19.681725 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.681699 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm"] Oct 31 00:43:19.684513 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:43:19.684481 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a22d4f_183e_4d55_b8f5_71c4d30736bb.slice/crio-7c21d27e68b2347acb53c28039264b9b94a53c76bcd878a1a30afe8f1a9b1e13 WatchSource:0}: Error finding container 7c21d27e68b2347acb53c28039264b9b94a53c76bcd878a1a30afe8f1a9b1e13: Status 404 returned error can't find the container with id 7c21d27e68b2347acb53c28039264b9b94a53c76bcd878a1a30afe8f1a9b1e13 Oct 31 00:43:19.842459 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.842423 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:19.918533 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.918500 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:19.923814 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.923786 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld"] Oct 31 00:43:19.927384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.927326 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" Oct 31 00:43:19.940206 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.940129 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld"] Oct 31 00:43:19.940877 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.940841 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f99ac0\"" Oct 31 00:43:19.940877 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:19.940841 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f99ac0-dockercfg-nt8gv\"" Oct 31 00:43:20.029267 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.029018 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76bbc2e-d9b0-4ac4-a09d-91b609760c16-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld\" (UID: \"f76bbc2e-d9b0-4ac4-a09d-91b609760c16\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" Oct 31 00:43:20.029267 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.029158 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2gqk\" (UniqueName: \"kubernetes.io/projected/f76bbc2e-d9b0-4ac4-a09d-91b609760c16-kube-api-access-m2gqk\") pod \"payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld\" (UID: \"f76bbc2e-d9b0-4ac4-a09d-91b609760c16\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" Oct 31 00:43:20.131710 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.130630 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2gqk\" (UniqueName: \"kubernetes.io/projected/f76bbc2e-d9b0-4ac4-a09d-91b609760c16-kube-api-access-m2gqk\") pod \"payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld\" (UID: \"f76bbc2e-d9b0-4ac4-a09d-91b609760c16\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" Oct 31 00:43:20.131710 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.130709 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76bbc2e-d9b0-4ac4-a09d-91b609760c16-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld\" (UID: \"f76bbc2e-d9b0-4ac4-a09d-91b609760c16\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" Oct 31 00:43:20.131710 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.131431 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76bbc2e-d9b0-4ac4-a09d-91b609760c16-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld\" (UID: \"f76bbc2e-d9b0-4ac4-a09d-91b609760c16\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" Oct 31 00:43:20.143917 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.143885 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2gqk\" (UniqueName: \"kubernetes.io/projected/f76bbc2e-d9b0-4ac4-a09d-91b609760c16-kube-api-access-m2gqk\") pod \"payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld\" (UID: \"f76bbc2e-d9b0-4ac4-a09d-91b609760c16\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" Oct 31 00:43:20.253661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.253445 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" Oct 31 00:43:20.283918 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.283869 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:20.450921 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.450824 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" event={"ID":"a1a22d4f-183e-4d55-b8f5-71c4d30736bb","Type":"ContainerStarted","Data":"7c21d27e68b2347acb53c28039264b9b94a53c76bcd878a1a30afe8f1a9b1e13"} Oct 31 00:43:20.561117 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.561029 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld"] Oct 31 00:43:20.566759 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:43:20.566716 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76bbc2e_d9b0_4ac4_a09d_91b609760c16.slice/crio-3996541fd0b7239e105f18d414e70e3bfb760891a0485cb6cc0152286cd7ddee WatchSource:0}: Error finding container 3996541fd0b7239e105f18d414e70e3bfb760891a0485cb6cc0152286cd7ddee: Status 404 returned error can't find the container with id 3996541fd0b7239e105f18d414e70e3bfb760891a0485cb6cc0152286cd7ddee Oct 31 00:43:20.595624 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.595584 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:20.645653 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.644059 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs"] Oct 31 00:43:20.649162 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.649135 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" Oct 31 00:43:20.665839 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.665769 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs"] Oct 31 00:43:20.677963 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.677886 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:20.737107 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.737053 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-1d4a7e\"" Oct 31 00:43:20.737603 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.737411 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39d45bb-7e82-4c44-8038-491cf125e145-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs\" (UID: \"d39d45bb-7e82-4c44-8038-491cf125e145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" Oct 31 00:43:20.737603 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.737446 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fngq6\" (UniqueName: \"kubernetes.io/projected/d39d45bb-7e82-4c44-8038-491cf125e145-kube-api-access-fngq6\") pod \"payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs\" (UID: \"d39d45bb-7e82-4c44-8038-491cf125e145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" Oct 31 00:43:20.838662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.838515 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39d45bb-7e82-4c44-8038-491cf125e145-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs\" (UID: \"d39d45bb-7e82-4c44-8038-491cf125e145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" Oct 31 00:43:20.838662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.838570 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fngq6\" (UniqueName: \"kubernetes.io/projected/d39d45bb-7e82-4c44-8038-491cf125e145-kube-api-access-fngq6\") pod \"payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs\" (UID: \"d39d45bb-7e82-4c44-8038-491cf125e145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" Oct 31 00:43:20.839611 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.839588 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39d45bb-7e82-4c44-8038-491cf125e145-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs\" (UID: \"d39d45bb-7e82-4c44-8038-491cf125e145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" Oct 31 00:43:20.867672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.861145 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fngq6\" (UniqueName: \"kubernetes.io/projected/d39d45bb-7e82-4c44-8038-491cf125e145-kube-api-access-fngq6\") pod \"payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs\" (UID: \"d39d45bb-7e82-4c44-8038-491cf125e145\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" Oct 31 00:43:20.972096 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:20.971812 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" Oct 31 00:43:21.184853 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:21.184664 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:21.277108 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:21.277008 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:21.290589 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:21.290551 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs"] Oct 31 00:43:21.423112 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:21.422999 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:21.469277 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:21.469180 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" event={"ID":"d39d45bb-7e82-4c44-8038-491cf125e145","Type":"ContainerStarted","Data":"2305eb4f24f97b299d9dafe7bb405f31100e69e8cd30630e9c2f868b05f979fd"} Oct 31 00:43:21.471944 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:21.471200 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" event={"ID":"f76bbc2e-d9b0-4ac4-a09d-91b609760c16","Type":"ContainerStarted","Data":"3996541fd0b7239e105f18d414e70e3bfb760891a0485cb6cc0152286cd7ddee"} Oct 31 00:43:22.166665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:22.158138 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:22.261258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:22.261179 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:22.421869 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:22.421772 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:22.481610 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:22.481577 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" event={"ID":"6b03043a-75b4-404a-800f-d71f33a04108","Type":"ContainerStarted","Data":"7544849631b0f15a08d6e0b17f5c84594a76b78e0111356aebde942dad29b6f4"} Oct 31 00:43:22.481929 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:22.481865 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" Oct 31 00:43:22.499978 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:22.499918 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" podStartSLOduration=1.738068815 podStartE2EDuration="5.499896902s" totalImagesPullingTime="3.761828087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:22.499187682 +0000 UTC m=+598.528368073" watchObservedRunningTime="2026-10-31 00:43:22.499896902 +0000 UTC m=+598.529077293" Oct 31 00:43:23.042824 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.042769 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:23.487358 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.487320 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" event={"ID":"d39d45bb-7e82-4c44-8038-491cf125e145","Type":"ContainerStarted","Data":"985f3bfd29a7b8f1705cfb0b397b347962bdf17500f388c8ecf5f3fae78123cc"} Oct 31 00:43:23.487843 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.487386 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" Oct 31 00:43:23.488930 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.488905 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" event={"ID":"7dd65b23-b72a-4eb4-971f-5f218554d0f2","Type":"ContainerStarted","Data":"8134f364768d468c6959d1f9c3bf701e1f827eb2c4c57268a3872f0688d0e76e"} Oct 31 00:43:23.489059 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.488947 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" Oct 31 00:43:23.490302 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.490282 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" event={"ID":"fd5bf7df-a3e9-4de3-ac84-e258676c6f92","Type":"ContainerStarted","Data":"e775d68e1dcfbef4cab975ccba7835d1894fed65283422e6ce56825a368ec5e0"} Oct 31 00:43:23.490405 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.490333 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" Oct 31 00:43:23.491776 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.491755 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" event={"ID":"a1a22d4f-183e-4d55-b8f5-71c4d30736bb","Type":"ContainerStarted","Data":"d055e0dd2ce7e62f3e3a0436b35d38ee279a102d2476e730b5f6a5bbe4d56b65"} Oct 31 00:43:23.491857 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.491803 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" Oct 31 00:43:23.493524 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.493502 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" event={"ID":"f76bbc2e-d9b0-4ac4-a09d-91b609760c16","Type":"ContainerStarted","Data":"e8b4494e095cdf344c1f883571ca176a2ed607955140e47b6e1b40963ffb0d6a"} Oct 31 00:43:23.493669 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.493657 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" Oct 31 00:43:23.506195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.506150 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" podStartSLOduration=1.515476375 podStartE2EDuration="3.506135255s" totalImagesPullingTime="1.99065888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:23.505534198 +0000 UTC m=+599.534714600" watchObservedRunningTime="2026-10-31 00:43:23.506135255 +0000 UTC m=+599.535315646" Oct 31 00:43:23.522395 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.522349 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" podStartSLOduration=1.6801179400000001 podStartE2EDuration="4.522336278s" totalImagesPullingTime="2.842218338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:23.520524889 +0000 UTC m=+599.549705301" watchObservedRunningTime="2026-10-31 00:43:23.522336278 +0000 UTC m=+599.551516669" Oct 31 00:43:23.541229 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.541183 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" podStartSLOduration=2.13121237 podStartE2EDuration="4.541170363s" totalImagesPullingTime="2.409957993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:23.539677111 +0000 UTC m=+599.568857492" watchObservedRunningTime="2026-10-31 00:43:23.541170363 +0000 UTC m=+599.570350752" Oct 31 00:43:23.557512 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.557415 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" podStartSLOduration=1.615314926 podStartE2EDuration="5.557404243s" totalImagesPullingTime="3.942089317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:23.556400121 +0000 UTC m=+599.585580518" watchObservedRunningTime="2026-10-31 00:43:23.557404243 +0000 UTC m=+599.586584632" Oct 31 00:43:23.572984 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.572941 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" podStartSLOduration=1.607716232 podStartE2EDuration="5.572928583s" totalImagesPullingTime="3.965212351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:23.571929672 +0000 UTC m=+599.601110067" watchObservedRunningTime="2026-10-31 00:43:23.572928583 +0000 UTC m=+599.602108972" Oct 31 00:43:23.823571 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.823484 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph"] Oct 31 00:43:23.827244 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.827220 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:43:23.836876 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.836847 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xds7h\"" Oct 31 00:43:23.837005 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.836970 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 00:43:23.837068 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.837007 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-f99ac0-serving-cert\"" Oct 31 00:43:23.837068 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.837023 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:43:23.837171 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.837111 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:43:23.837331 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.837312 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph"] Oct 31 00:43:23.865115 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.865080 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:23.889162 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.889134 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638ee841-3ada-4014-8841-f9da201ea000-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph\" (UID: \"638ee841-3ada-4014-8841-f9da201ea000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:43:23.889331 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.889181 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf5v2\" (UniqueName: \"kubernetes.io/projected/638ee841-3ada-4014-8841-f9da201ea000-kube-api-access-vf5v2\") pod \"maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph\" (UID: \"638ee841-3ada-4014-8841-f9da201ea000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:43:23.889331 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.889263 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/638ee841-3ada-4014-8841-f9da201ea000-metrics-certs\") pod \"maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph\" (UID: \"638ee841-3ada-4014-8841-f9da201ea000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:43:23.889331 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.889317 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638ee841-3ada-4014-8841-f9da201ea000-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph\" (UID: \"638ee841-3ada-4014-8841-f9da201ea000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:43:23.968975 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.968934 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:23.990206 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.990167 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf5v2\" (UniqueName: \"kubernetes.io/projected/638ee841-3ada-4014-8841-f9da201ea000-kube-api-access-vf5v2\") pod \"maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph\" (UID: \"638ee841-3ada-4014-8841-f9da201ea000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:43:23.990411 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.990237 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/638ee841-3ada-4014-8841-f9da201ea000-metrics-certs\") pod \"maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph\" (UID: \"638ee841-3ada-4014-8841-f9da201ea000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:43:23.990411 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.990288 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638ee841-3ada-4014-8841-f9da201ea000-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph\" (UID: \"638ee841-3ada-4014-8841-f9da201ea000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:43:23.990411 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.990357 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638ee841-3ada-4014-8841-f9da201ea000-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph\" (UID: \"638ee841-3ada-4014-8841-f9da201ea000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:43:23.991408 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.991383 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638ee841-3ada-4014-8841-f9da201ea000-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph\" (UID: \"638ee841-3ada-4014-8841-f9da201ea000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:43:23.993470 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.993407 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/638ee841-3ada-4014-8841-f9da201ea000-metrics-certs\") pod \"maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph\" (UID: \"638ee841-3ada-4014-8841-f9da201ea000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:43:23.994263 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:23.994239 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638ee841-3ada-4014-8841-f9da201ea000-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph\" (UID: \"638ee841-3ada-4014-8841-f9da201ea000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:43:24.004082 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:24.004058 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf5v2\" (UniqueName: \"kubernetes.io/projected/638ee841-3ada-4014-8841-f9da201ea000-kube-api-access-vf5v2\") pod \"maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph\" (UID: \"638ee841-3ada-4014-8841-f9da201ea000\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:43:24.146471 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:24.146394 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xds7h\"" Oct 31 00:43:24.153355 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:24.153324 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:43:24.438103 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:24.438074 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph"] Oct 31 00:43:24.440302 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:43:24.440271 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638ee841_3ada_4014_8841_f9da201ea000.slice/crio-92993e7d62fc191a4de90ee0dd999a73aa199c0bc655d7bbbabc7e54ceecf03c WatchSource:0}: Error finding container 92993e7d62fc191a4de90ee0dd999a73aa199c0bc655d7bbbabc7e54ceecf03c: Status 404 returned error can't find the container with id 92993e7d62fc191a4de90ee0dd999a73aa199c0bc655d7bbbabc7e54ceecf03c Oct 31 00:43:24.498727 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:24.498689 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" event={"ID":"638ee841-3ada-4014-8841-f9da201ea000","Type":"ContainerStarted","Data":"92993e7d62fc191a4de90ee0dd999a73aa199c0bc655d7bbbabc7e54ceecf03c"} Oct 31 00:43:24.673270 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:24.673229 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:24.777054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:24.777015 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:24.880792 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:24.880759 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:25.433060 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.433023 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6"] Oct 31 00:43:25.437118 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.437088 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:43:25.453117 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.450958 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-b433e6-serving-cert\"" Oct 31 00:43:25.453571 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.453546 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6"] Oct 31 00:43:25.481711 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.481678 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:25.502626 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.502591 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6\" (UID: \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:43:25.503169 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.502656 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rggkt\" (UniqueName: \"kubernetes.io/projected/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-kube-api-access-rggkt\") pod \"maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6\" (UID: \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:43:25.503169 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.502808 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6\" (UID: \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:43:25.503169 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.502850 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-metrics-certs\") pod \"maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6\" (UID: \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:43:25.553681 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.553611 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:25.604006 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.603966 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6\" (UID: \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:43:25.604179 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.604014 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rggkt\" (UniqueName: \"kubernetes.io/projected/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-kube-api-access-rggkt\") pod \"maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6\" (UID: \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:43:25.604179 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.604079 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6\" (UID: \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:43:25.604179 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.604100 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-metrics-certs\") pod \"maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6\" (UID: \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:43:25.605069 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.605021 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6\" (UID: \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:43:25.606877 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.606855 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6\" (UID: \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:43:25.606986 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.606930 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-metrics-certs\") pod \"maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6\" (UID: \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:43:25.617621 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.617588 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rggkt\" (UniqueName: \"kubernetes.io/projected/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-kube-api-access-rggkt\") pod \"maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6\" (UID: \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:43:25.656653 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.656592 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:25.758870 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:25.758835 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:43:26.043067 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:26.043036 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6"] Oct 31 00:43:26.045536 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:43:26.045510 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1e90c3_947d_41f2_a4bb_fca7f4b54c40.slice/crio-f8f88f723be8c1e3a0f91537ee71413c3c8f512ae106f82b045c4494e4bb0a00 WatchSource:0}: Error finding container f8f88f723be8c1e3a0f91537ee71413c3c8f512ae106f82b045c4494e4bb0a00: Status 404 returned error can't find the container with id f8f88f723be8c1e3a0f91537ee71413c3c8f512ae106f82b045c4494e4bb0a00 Oct 31 00:43:26.278398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:26.278361 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:26.509451 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:26.509411 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" event={"ID":"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40","Type":"ContainerStarted","Data":"f8f88f723be8c1e3a0f91537ee71413c3c8f512ae106f82b045c4494e4bb0a00"} Oct 31 00:43:27.333480 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:27.333442 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:27.517729 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:27.517689 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" event={"ID":"638ee841-3ada-4014-8841-f9da201ea000","Type":"ContainerStarted","Data":"7012a352ea43e9db0648a6f00bda576fe5333c4fd993ba9d93ad5783dbc6fe7d"} Oct 31 00:43:27.518294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:27.518278 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:43:27.541328 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:27.541237 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" podStartSLOduration=1.699095587 podStartE2EDuration="4.54122401s" totalImagesPullingTime="2.842128423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:27.538909705 +0000 UTC m=+603.568090096" watchObservedRunningTime="2026-10-31 00:43:27.54122401 +0000 UTC m=+603.570404400" Oct 31 00:43:28.469600 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:28.469568 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:28.541269 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:28.541230 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" event={"ID":"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40","Type":"ContainerStarted","Data":"b4ab2e84369bc88094854fa22ed3bfd8d12102fde92fa14544d43aadc58d21fd"} Oct 31 00:43:28.541772 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:28.541329 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:43:28.560516 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:28.560461 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" podStartSLOduration=2.130838035 podStartE2EDuration="3.560444377s" totalImagesPullingTime="1.429606342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:28.559231507 +0000 UTC m=+604.588411898" watchObservedRunningTime="2026-10-31 00:43:28.560444377 +0000 UTC m=+604.589624767" Oct 31 00:43:29.328830 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:29.328795 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:30.264835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:30.264794 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:31.041148 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:31.041113 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:31.140312 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:31.140273 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:31.529124 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:31.529089 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:31.664859 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:31.664827 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:31.763619 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:31.763588 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:31.863740 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:31.863606 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:32.027236 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:32.027199 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:33.015913 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.015875 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:33.305999 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.305921 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d"] Oct 31 00:43:33.309732 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.309711 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.318168 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.318140 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7f1496\"/\"openshift-service-ca.crt\"" Oct 31 00:43:33.318296 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.318270 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7f1496\"/\"kube-root-ca.crt\"" Oct 31 00:43:33.319462 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.319435 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d"] Oct 31 00:43:33.334044 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.333998 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7f1496\"/\"simulator-w1-7f1496-kserve-self-signed-certs\"" Oct 31 00:43:33.334521 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.334499 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7f1496\"/\"default-dockercfg-zqzc6\"" Oct 31 00:43:33.347983 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.347935 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:33.382432 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.382390 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-tmp-dir\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.382612 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.382445 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-home\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.382612 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.382510 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-dshm\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.382612 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.382576 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/804eee3f-86ba-4511-a9c2-279edb91cf38-tls-certs\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.382612 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.382603 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt9d9\" (UniqueName: \"kubernetes.io/projected/804eee3f-86ba-4511-a9c2-279edb91cf38-kube-api-access-zt9d9\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.382793 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.382730 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-model-cache\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.490776 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.490699 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-tmp-dir\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.491025 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.491003 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-home\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.491131 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.491117 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-dshm\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.491230 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.491215 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/804eee3f-86ba-4511-a9c2-279edb91cf38-tls-certs\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.491330 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.491316 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt9d9\" (UniqueName: \"kubernetes.io/projected/804eee3f-86ba-4511-a9c2-279edb91cf38-kube-api-access-zt9d9\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.491500 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.491484 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-model-cache\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.492162 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.492136 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-model-cache\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.492301 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.491321 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-tmp-dir\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.493083 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.493052 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-home\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.497868 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.497798 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/804eee3f-86ba-4511-a9c2-279edb91cf38-tls-certs\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.498014 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.497897 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-dshm\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.503783 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.503752 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" Oct 31 00:43:33.513210 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.513176 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt9d9\" (UniqueName: \"kubernetes.io/projected/804eee3f-86ba-4511-a9c2-279edb91cf38-kube-api-access-zt9d9\") pod \"simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.630213 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.630118 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:33.910154 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:33.910038 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d"] Oct 31 00:43:33.912552 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:43:33.912524 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod804eee3f_86ba_4511_a9c2_279edb91cf38.slice/crio-402dbebe6d0843f29deacf94c0a6b9fe3342a486b6d22a570968cbc4cf406ffc WatchSource:0}: Error finding container 402dbebe6d0843f29deacf94c0a6b9fe3342a486b6d22a570968cbc4cf406ffc: Status 404 returned error can't find the container with id 402dbebe6d0843f29deacf94c0a6b9fe3342a486b6d22a570968cbc4cf406ffc Oct 31 00:43:34.105754 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:34.105724 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:34.390145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:34.390102 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:34.502232 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:34.502199 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" Oct 31 00:43:34.504716 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:34.504665 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" Oct 31 00:43:34.505380 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:34.505351 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" Oct 31 00:43:34.505496 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:34.505467 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" Oct 31 00:43:34.506243 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:34.506220 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" Oct 31 00:43:34.556037 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:34.556006 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:43:34.579415 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:34.579350 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" event={"ID":"804eee3f-86ba-4511-a9c2-279edb91cf38","Type":"ContainerStarted","Data":"402dbebe6d0843f29deacf94c0a6b9fe3342a486b6d22a570968cbc4cf406ffc"} Oct 31 00:43:35.330823 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:35.330797 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:35.556798 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:35.556726 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:43:35.584839 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:35.584807 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" event={"ID":"804eee3f-86ba-4511-a9c2-279edb91cf38","Type":"ContainerStarted","Data":"7a23d0c2b116f2e5ef532751a0b422ea505f03a538521e0f512081246a29cc6a"} Oct 31 00:43:35.631036 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:35.631003 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:35.687328 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:35.687293 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:36.195207 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:36.195175 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:36.325960 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:36.325919 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:36.427025 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:36.426983 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:36.525614 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:36.525577 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:36.691755 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:36.691713 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:37.326329 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:37.326299 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:37.454488 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:37.454455 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:37.663113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:37.663024 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:38.783620 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:38.783588 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:39.928532 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:39.928504 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:40.982711 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:40.982681 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:41.307684 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:41.307581 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:41.938506 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:41.938474 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:42.042305 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.042270 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:42.119245 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.119193 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:42.252427 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.252333 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podStartSLOduration=7.886736032 podStartE2EDuration="9.252312375s" totalImagesPullingTime="1.365576343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:35.606333233 +0000 UTC m=+611.635513626" watchObservedRunningTime="2026-10-31 00:43:42.252312375 +0000 UTC m=+618.281492767" Oct 31 00:43:42.254393 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.254054 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz"] Oct 31 00:43:42.258411 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.258391 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.269985 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.269954 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz"] Oct 31 00:43:42.275700 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.275669 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-47f370\"/\"kube-root-ca.crt\"" Oct 31 00:43:42.276954 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.275742 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-47f370\"/\"simulator-w6-47f370-kserve-self-signed-certs\"" Oct 31 00:43:42.277042 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.277005 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-47f370\"/\"openshift-service-ca.crt\"" Oct 31 00:43:42.277104 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.277077 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-47f370\"/\"default-dockercfg-42hms\"" Oct 31 00:43:42.307906 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.307866 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:42.377012 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.376971 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-model-cache\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.377187 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.377031 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-dshm\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.377187 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.377088 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2ss5\" (UniqueName: \"kubernetes.io/projected/bf5f1905-6c45-4ea7-8871-5676c27e80a7-kube-api-access-j2ss5\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.377327 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.377225 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5f1905-6c45-4ea7-8871-5676c27e80a7-tls-certs\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.377327 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.377273 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-tmp-dir\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.377434 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.377346 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-home\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.478547 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.478508 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5f1905-6c45-4ea7-8871-5676c27e80a7-tls-certs\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.478711 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.478553 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-tmp-dir\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.478785 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.478750 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-home\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.478841 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.478794 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-model-cache\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.478895 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.478860 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-dshm\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.478895 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.478886 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2ss5\" (UniqueName: \"kubernetes.io/projected/bf5f1905-6c45-4ea7-8871-5676c27e80a7-kube-api-access-j2ss5\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.479006 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.478987 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-tmp-dir\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.479131 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.479111 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-home\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.479242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.479219 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-model-cache\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.480994 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.480967 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-dshm\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.481076 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.481045 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5f1905-6c45-4ea7-8871-5676c27e80a7-tls-certs\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.493254 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.493227 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2ss5\" (UniqueName: \"kubernetes.io/projected/bf5f1905-6c45-4ea7-8871-5676c27e80a7-kube-api-access-j2ss5\") pod \"simulator-w6-47f370-kserve-7895c56c66-nr5jz\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.571773 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.571688 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:42.863497 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:42.863469 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz"] Oct 31 00:43:42.865589 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:43:42.865559 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf5f1905_6c45_4ea7_8871_5676c27e80a7.slice/crio-4643e02c283c90850b676f1d3d60d15d3444f3579746492672c79a71b83e7224 WatchSource:0}: Error finding container 4643e02c283c90850b676f1d3d60d15d3444f3579746492672c79a71b83e7224: Status 404 returned error can't find the container with id 4643e02c283c90850b676f1d3d60d15d3444f3579746492672c79a71b83e7224 Oct 31 00:43:43.300242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:43.300209 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:43.626042 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:43.625960 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" event={"ID":"bf5f1905-6c45-4ea7-8871-5676c27e80a7","Type":"ContainerStarted","Data":"bf44831817b786fce85a1ba53f831f4652fc9c845ea094494105f3262f12cc26"} Oct 31 00:43:43.626042 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:43.625999 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" event={"ID":"bf5f1905-6c45-4ea7-8871-5676c27e80a7","Type":"ContainerStarted","Data":"4643e02c283c90850b676f1d3d60d15d3444f3579746492672c79a71b83e7224"} Oct 31 00:43:43.630502 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:43.630472 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:43.643249 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:43.643226 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:43.649975 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:43.649931 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podStartSLOduration=1.649919491 podStartE2EDuration="1.649919491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:43.647993092 +0000 UTC m=+619.677173485" watchObservedRunningTime="2026-10-31 00:43:43.649919491 +0000 UTC m=+619.679099887" Oct 31 00:43:44.275708 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:44.275677 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:44.572478 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:44.572396 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:44.642993 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:44.642946 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:43:44.681496 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:44.681462 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:45.375908 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:45.375874 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:46.374223 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:46.374164 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:47.070688 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:47.070648 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:47.162729 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:47.162692 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:47.464724 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:47.464612 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:47.547730 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:43:47.547693 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:43:47.714129 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:47.714093 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:47.810404 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:47.810367 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:47.964945 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:47.964897 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:48.135342 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:48.135259 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:48.999129 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:48.999090 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:49.680120 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.680084 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc"] Oct 31 00:43:49.705924 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.705891 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc"] Oct 31 00:43:49.706131 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.706033 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.710450 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.710424 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-44bc67\"/\"openshift-service-ca.crt\"" Oct 31 00:43:49.710650 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.710426 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-44bc67\"/\"kube-root-ca.crt\"" Oct 31 00:43:49.717066 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.717037 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-44bc67\"/\"simulator-w5-44bc67-kserve-self-signed-certs\"" Oct 31 00:43:49.717222 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.717100 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-44bc67\"/\"default-dockercfg-kvd55\"" Oct 31 00:43:49.718489 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.718460 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:49.846613 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.846578 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:49.848667 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.848623 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-home\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.848855 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.848726 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-dshm\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.848855 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.848748 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g66sl\" (UniqueName: \"kubernetes.io/projected/886c14ce-3e7b-491b-af66-4a7c9b119861-kube-api-access-g66sl\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.848979 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.848902 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-model-cache\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.848979 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.848946 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/886c14ce-3e7b-491b-af66-4a7c9b119861-tls-certs\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.849089 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.848988 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-tmp-dir\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.950060 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.949962 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-home\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.950060 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.950019 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-dshm\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.950060 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.950041 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g66sl\" (UniqueName: \"kubernetes.io/projected/886c14ce-3e7b-491b-af66-4a7c9b119861-kube-api-access-g66sl\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.950353 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.950114 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-model-cache\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.950353 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.950148 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/886c14ce-3e7b-491b-af66-4a7c9b119861-tls-certs\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.950353 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.950298 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-tmp-dir\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.950516 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.950421 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-home\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.950573 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.950515 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-model-cache\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.950673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.950654 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-tmp-dir\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.952413 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.952386 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-dshm\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.952730 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.952714 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/886c14ce-3e7b-491b-af66-4a7c9b119861-tls-certs\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:49.963021 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:49.962997 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g66sl\" (UniqueName: \"kubernetes.io/projected/886c14ce-3e7b-491b-af66-4a7c9b119861-kube-api-access-g66sl\") pod \"simulator-w5-44bc67-kserve-6c77894df6-5kbvc\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:50.034114 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:50.034072 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:50.336867 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:50.336840 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc"] Oct 31 00:43:50.338557 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:43:50.338520 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod886c14ce_3e7b_491b_af66_4a7c9b119861.slice/crio-22f86af2ed1e2dce39795a107c2be459e6a9ff4550e52e5aff976d6c8a311ee3 WatchSource:0}: Error finding container 22f86af2ed1e2dce39795a107c2be459e6a9ff4550e52e5aff976d6c8a311ee3: Status 404 returned error can't find the container with id 22f86af2ed1e2dce39795a107c2be459e6a9ff4550e52e5aff976d6c8a311ee3 Oct 31 00:43:50.583999 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:50.583961 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:50.662581 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:50.662493 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" event={"ID":"886c14ce-3e7b-491b-af66-4a7c9b119861","Type":"ContainerStarted","Data":"8bd1b63dde235a63d9db6676edf058ab426120cb0d2630ecb3e0b7ac8febc12a"} Oct 31 00:43:50.662581 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:50.662536 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" event={"ID":"886c14ce-3e7b-491b-af66-4a7c9b119861","Type":"ContainerStarted","Data":"22f86af2ed1e2dce39795a107c2be459e6a9ff4550e52e5aff976d6c8a311ee3"} Oct 31 00:43:50.682179 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:50.682122 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podStartSLOduration=1.6821071399999998 podStartE2EDuration="1.68210714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:50.679958448 +0000 UTC m=+626.709138839" watchObservedRunningTime="2026-10-31 00:43:50.68210714 +0000 UTC m=+626.711287529" Oct 31 00:43:51.034530 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:51.034493 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:43:51.329284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:51.329203 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:52.058680 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:52.058627 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:52.572687 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:52.572618 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:52.585916 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:52.585887 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:52.682273 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:52.682244 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:43:52.925612 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:52.925516 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:54.276340 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:54.276305 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:54.671112 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:54.671024 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:55.699685 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:55.699650 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:56.031161 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.031127 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl"] Oct 31 00:43:56.035243 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.035213 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.044655 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.044606 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b433e6\"/\"simulator-w2-b433e6-kserve-self-signed-certs\"" Oct 31 00:43:56.054276 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.054239 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b433e6\"/\"default-dockercfg-8qm9p\"" Oct 31 00:43:56.058343 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.058316 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl"] Oct 31 00:43:56.060083 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.059617 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b433e6\"/\"openshift-service-ca.crt\"" Oct 31 00:43:56.060083 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.059657 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b433e6\"/\"kube-root-ca.crt\"" Oct 31 00:43:56.103900 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.103864 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:56.107969 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.107944 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-home\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.108095 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.107979 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-dshm\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.108095 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.107997 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-tmp-dir\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.108095 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.108023 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d9d315d-8875-4e45-8d34-2005ddd628d5-tls-certs\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.108196 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.108176 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-model-cache\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.108230 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.108215 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62mh5\" (UniqueName: \"kubernetes.io/projected/0d9d315d-8875-4e45-8d34-2005ddd628d5-kube-api-access-62mh5\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.209541 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.209486 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-home\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.209541 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.209536 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-dshm\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.209808 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.209554 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-tmp-dir\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.209808 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.209688 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d9d315d-8875-4e45-8d34-2005ddd628d5-tls-certs\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.209808 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.209747 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-model-cache\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.209808 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.209779 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62mh5\" (UniqueName: \"kubernetes.io/projected/0d9d315d-8875-4e45-8d34-2005ddd628d5-kube-api-access-62mh5\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.210004 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.209972 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-home\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.210061 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.210047 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-tmp-dir\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.210280 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.210250 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-model-cache\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.212087 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.212058 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-dshm\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.212705 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.212683 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d9d315d-8875-4e45-8d34-2005ddd628d5-tls-certs\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.225846 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.225812 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62mh5\" (UniqueName: \"kubernetes.io/projected/0d9d315d-8875-4e45-8d34-2005ddd628d5-kube-api-access-62mh5\") pod \"simulator-w2-b433e6-kserve-bb7846f8d-6lmbl\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.350831 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.350747 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:56.646280 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.646188 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl"] Oct 31 00:43:56.651140 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:43:56.651114 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d9d315d_8875_4e45_8d34_2005ddd628d5.slice/crio-c87e74c44069ac17e535af8c709f30400d7b331538372938413227380d8e632b WatchSource:0}: Error finding container c87e74c44069ac17e535af8c709f30400d7b331538372938413227380d8e632b: Status 404 returned error can't find the container with id c87e74c44069ac17e535af8c709f30400d7b331538372938413227380d8e632b Oct 31 00:43:56.693355 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.693322 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" event={"ID":"0d9d315d-8875-4e45-8d34-2005ddd628d5","Type":"ContainerStarted","Data":"c87e74c44069ac17e535af8c709f30400d7b331538372938413227380d8e632b"} Oct 31 00:43:56.796042 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:56.796007 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:57.662780 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:57.662715 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:57.717982 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:57.717941 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" event={"ID":"0d9d315d-8875-4e45-8d34-2005ddd628d5","Type":"ContainerStarted","Data":"1db6086e867eafebe67d526fded02d3390ff41791870007fb666a2e7f5e84546"} Oct 31 00:43:57.750033 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:57.749979 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podStartSLOduration=1.7499642199999998 podStartE2EDuration="1.74996422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:57.74882438 +0000 UTC m=+633.778004782" watchObservedRunningTime="2026-10-31 00:43:57.74996422 +0000 UTC m=+633.779144631" Oct 31 00:43:58.351350 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:58.351318 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:43:58.368842 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:58.368804 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:58.846544 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:58.846480 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:59.167336 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:43:59.167252 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:00.033967 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:00.033928 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:00.034793 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:00.034775 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:44:00.047675 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:00.047629 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:44:00.742582 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:00.742550 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:44:01.224168 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:01.224073 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:01.939365 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:01.939321 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:02.733861 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:02.733821 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:03.658423 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:03.658389 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:04.084569 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.084523 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p"] Oct 31 00:44:04.088657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.088625 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.099581 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.099011 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5c2a03\"/\"default-dockercfg-6cdqc\"" Oct 31 00:44:04.099581 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.099011 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5c2a03\"/\"openshift-service-ca.crt\"" Oct 31 00:44:04.099581 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.099406 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5c2a03\"/\"kube-root-ca.crt\"" Oct 31 00:44:04.099581 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.099530 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5c2a03\"/\"premium-w4-5c2a03-kserve-self-signed-certs\"" Oct 31 00:44:04.107541 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.107509 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p"] Oct 31 00:44:04.127488 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.127446 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:04.192980 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.192936 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvgvl\" (UniqueName: \"kubernetes.io/projected/63751bb9-e9b0-4c48-9887-2bf875a39d87-kube-api-access-cvgvl\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.193156 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.192991 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-dshm\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.193204 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.193184 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-tmp-dir\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.193239 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.193223 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-home\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.193374 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.193356 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63751bb9-e9b0-4c48-9887-2bf875a39d87-tls-certs\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.193418 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.193403 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-model-cache\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.294264 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.294222 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63751bb9-e9b0-4c48-9887-2bf875a39d87-tls-certs\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.294472 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.294272 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-model-cache\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.294472 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.294316 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvgvl\" (UniqueName: \"kubernetes.io/projected/63751bb9-e9b0-4c48-9887-2bf875a39d87-kube-api-access-cvgvl\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.294472 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.294338 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-dshm\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.294472 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.294427 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-tmp-dir\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.294472 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.294451 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-home\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.294827 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.294799 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-model-cache\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.294988 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.294962 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-tmp-dir\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.295061 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.294881 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-home\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.296796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.296770 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-dshm\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.296957 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.296936 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63751bb9-e9b0-4c48-9887-2bf875a39d87-tls-certs\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.304001 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.303977 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvgvl\" (UniqueName: \"kubernetes.io/projected/63751bb9-e9b0-4c48-9887-2bf875a39d87-kube-api-access-cvgvl\") pod \"premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.404878 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.404766 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:04.445934 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.445903 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:04.685448 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.685418 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p"] Oct 31 00:44:04.687303 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:44:04.687276 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63751bb9_e9b0_4c48_9887_2bf875a39d87.slice/crio-8e020df3319688a765ac1763a258e851ae1123edb69cd9cc3e7e1d26a32824d0 WatchSource:0}: Error finding container 8e020df3319688a765ac1763a258e851ae1123edb69cd9cc3e7e1d26a32824d0: Status 404 returned error can't find the container with id 8e020df3319688a765ac1763a258e851ae1123edb69cd9cc3e7e1d26a32824d0 Oct 31 00:44:04.752446 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:04.752404 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" event={"ID":"63751bb9-e9b0-4c48-9887-2bf875a39d87","Type":"ContainerStarted","Data":"8e020df3319688a765ac1763a258e851ae1123edb69cd9cc3e7e1d26a32824d0"} Oct 31 00:44:05.578879 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:05.578844 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:05.758359 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:05.758325 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" event={"ID":"63751bb9-e9b0-4c48-9887-2bf875a39d87","Type":"ContainerStarted","Data":"3c3bd621840c39b9b4513d91a54f8fb09e4d437e54e27be3e2e4966c3cb5c2ac"} Oct 31 00:44:05.779870 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:05.779819 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podStartSLOduration=1.7798052979999999 podStartE2EDuration="1.779805298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:05.779509265 +0000 UTC m=+641.808689655" watchObservedRunningTime="2026-10-31 00:44:05.779805298 +0000 UTC m=+641.808985688" Oct 31 00:44:06.351365 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:06.351324 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:44:06.356430 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:06.356395 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:06.364533 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:06.364501 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:44:06.405538 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:06.405495 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:06.774839 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:06.774810 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:44:07.140504 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:07.140392 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:07.941798 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:07.941761 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:09.003277 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:09.003222 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:10.232443 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:10.232412 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:10.336101 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:10.336069 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:11.144365 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:11.144321 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:11.844654 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:11.844603 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:12.660544 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.660512 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:12.784560 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.784516 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj"] Oct 31 00:44:12.788395 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.788370 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.797847 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.797812 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-44bc67\"/\"premium-w5-44bc67-kserve-self-signed-certs\"" Oct 31 00:44:12.799287 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.799265 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj"] Oct 31 00:44:12.813981 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.813950 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:12.879801 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.879757 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-tmp-dir\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.880281 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.879901 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-home\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.880281 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.879961 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a6c2be-3886-4f44-a65a-7eef406c0a45-tls-certs\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.880281 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.880087 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz9x4\" (UniqueName: \"kubernetes.io/projected/b2a6c2be-3886-4f44-a65a-7eef406c0a45-kube-api-access-xz9x4\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.880281 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.880143 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-dshm\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.880281 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.880173 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-model-cache\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.980938 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.980904 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-home\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.981152 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.980966 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a6c2be-3886-4f44-a65a-7eef406c0a45-tls-certs\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.981152 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.981015 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz9x4\" (UniqueName: \"kubernetes.io/projected/b2a6c2be-3886-4f44-a65a-7eef406c0a45-kube-api-access-xz9x4\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.981152 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.981061 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-dshm\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.981152 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.981086 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-model-cache\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.981408 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.981376 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-home\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.981555 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.981427 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-tmp-dir\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.981555 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.981426 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-model-cache\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.981702 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.981673 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-tmp-dir\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.983717 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.983696 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a6c2be-3886-4f44-a65a-7eef406c0a45-tls-certs\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.983926 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.983901 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-dshm\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:12.994450 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:12.994421 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz9x4\" (UniqueName: \"kubernetes.io/projected/b2a6c2be-3886-4f44-a65a-7eef406c0a45-kube-api-access-xz9x4\") pod \"premium-w5-44bc67-kserve-599494fcf7-hgztj\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:13.104417 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:13.104373 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:13.336369 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:13.336290 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:13.378123 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:13.378058 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj"] Oct 31 00:44:13.381404 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:44:13.381369 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2a6c2be_3886_4f44_a65a_7eef406c0a45.slice/crio-2d8bcd0825778a5010bddf6d5280548f04105d67e8148fc650aff1489fdb1de3 WatchSource:0}: Error finding container 2d8bcd0825778a5010bddf6d5280548f04105d67e8148fc650aff1489fdb1de3: Status 404 returned error can't find the container with id 2d8bcd0825778a5010bddf6d5280548f04105d67e8148fc650aff1489fdb1de3 Oct 31 00:44:13.799052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:13.799022 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" event={"ID":"b2a6c2be-3886-4f44-a65a-7eef406c0a45","Type":"ContainerStarted","Data":"cdd211cf9b890a440e36df4f5b43157b6407fcd6d3929898f39b1efe92400f35"} Oct 31 00:44:13.799218 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:13.799062 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" event={"ID":"b2a6c2be-3886-4f44-a65a-7eef406c0a45","Type":"ContainerStarted","Data":"2d8bcd0825778a5010bddf6d5280548f04105d67e8148fc650aff1489fdb1de3"} Oct 31 00:44:13.819036 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:13.818967 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podStartSLOduration=1.8189489330000002 podStartE2EDuration="1.818948933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:13.818320319 +0000 UTC m=+649.847500710" watchObservedRunningTime="2026-10-31 00:44:13.818948933 +0000 UTC m=+649.848129323" Oct 31 00:44:14.060686 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:14.060554 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:14.104794 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:14.104762 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:14.405820 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:14.405731 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:14.418938 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:14.418911 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:14.805399 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:14.805291 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:14.815574 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:14.815550 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:44:14.998225 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:14.998177 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:15.783437 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:15.783401 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:16.560469 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:16.560430 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:17.069545 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:17.069501 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:17.454715 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:17.454580 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:17.548772 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:44:17.548729 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:44:18.393834 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:18.393803 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:19.105979 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.105943 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz"] Oct 31 00:44:19.109800 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.109779 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.116176 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.116150 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b433e6\"/\"premium-w2-b433e6-kserve-self-signed-certs\"" Oct 31 00:44:19.118390 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.118366 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz"] Oct 31 00:44:19.145832 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.145797 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:19.148985 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.148960 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxtf2\" (UniqueName: \"kubernetes.io/projected/4298cde5-e51e-42f5-924d-0c2224a1f57e-kube-api-access-qxtf2\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.149100 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.149021 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4298cde5-e51e-42f5-924d-0c2224a1f57e-tls-certs\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.149166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.149142 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-home\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.149221 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.149195 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-model-cache\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.149280 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.149224 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-dshm\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.149423 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.149401 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-tmp-dir\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.252660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.250194 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-tmp-dir\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.252660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.250275 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxtf2\" (UniqueName: \"kubernetes.io/projected/4298cde5-e51e-42f5-924d-0c2224a1f57e-kube-api-access-qxtf2\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.252660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.250345 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4298cde5-e51e-42f5-924d-0c2224a1f57e-tls-certs\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.252660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.250386 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-home\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.252660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.250420 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-model-cache\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.252660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.250448 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-dshm\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.252660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.251810 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-home\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.252660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.252181 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-model-cache\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.252660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.252400 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-tmp-dir\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.253524 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.253448 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-dshm\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.254628 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.254063 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4298cde5-e51e-42f5-924d-0c2224a1f57e-tls-certs\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.275661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.266002 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxtf2\" (UniqueName: \"kubernetes.io/projected/4298cde5-e51e-42f5-924d-0c2224a1f57e-kube-api-access-qxtf2\") pod \"premium-w2-b433e6-kserve-7d7d48597c-lddlz\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.310650 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.310593 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:19.428352 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.428267 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:19.700369 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.700345 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz"] Oct 31 00:44:19.702572 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:44:19.702545 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4298cde5_e51e_42f5_924d_0c2224a1f57e.slice/crio-fe5219450cffa3a186a29335ee8ac1de16f0b9d0ec45cbd42e20fb1b2283145b WatchSource:0}: Error finding container fe5219450cffa3a186a29335ee8ac1de16f0b9d0ec45cbd42e20fb1b2283145b: Status 404 returned error can't find the container with id fe5219450cffa3a186a29335ee8ac1de16f0b9d0ec45cbd42e20fb1b2283145b Oct 31 00:44:19.828665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.828617 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" event={"ID":"4298cde5-e51e-42f5-924d-0c2224a1f57e","Type":"ContainerStarted","Data":"9c2b9907c8b36a44e201bf4ecc634088534a7d3564c31aad87df5fd19e7152d7"} Oct 31 00:44:19.828665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.828669 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" event={"ID":"4298cde5-e51e-42f5-924d-0c2224a1f57e","Type":"ContainerStarted","Data":"fe5219450cffa3a186a29335ee8ac1de16f0b9d0ec45cbd42e20fb1b2283145b"} Oct 31 00:44:19.849456 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:19.849379 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podStartSLOduration=0.8493656 podStartE2EDuration="849.3656ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:19.847475973 +0000 UTC m=+655.876656355" watchObservedRunningTime="2026-10-31 00:44:19.8493656 +0000 UTC m=+655.878545989" Oct 31 00:44:20.068316 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:20.068283 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:20.429208 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:20.429133 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:20.755389 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:20.755348 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:21.536647 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:21.536604 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:22.371033 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:22.371003 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:23.104974 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:23.104941 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:23.117528 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:23.117506 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:23.464989 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:23.464912 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:23.778707 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:23.778677 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:23.886130 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:23.886090 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:44:24.456861 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:24.456822 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:25.373145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:25.373107 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:26.263784 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:26.263750 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:27.062899 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:27.062867 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:27.852935 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:27.852900 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:28.563410 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:28.563369 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:29.429363 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:29.429317 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:29.442294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:29.442262 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:29.476278 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:29.476243 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:29.892289 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:29.892260 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:44:30.185051 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:30.184954 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:31.023061 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:31.023024 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:31.308194 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:31.308119 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:31.759555 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:31.759508 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:32.546735 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:32.546676 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:33.368419 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:33.368380 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:34.782894 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:34.782856 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:36.568669 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:36.568563 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:38.402764 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:38.402723 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:39.498688 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:39.498656 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:40.385686 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:40.385621 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:41.205864 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:41.205827 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:41.964065 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:41.964030 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:43.162748 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:43.162711 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:44.172039 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:44.172002 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:45.428603 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:45.428562 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:46.590284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:46.590244 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:47.542105 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:44:47.542064 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:44:47.786554 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:47.786519 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:48.712696 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:48.712657 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:49.586776 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:49.586741 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:50.306008 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:50.305971 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:51.077114 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:51.077079 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:51.870171 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:51.870140 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:52.540592 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:52.540562 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:53.438205 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:53.438170 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:54.121703 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:54.121670 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:54.739343 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:54.739312 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:55.445823 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:55.445792 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:56.092494 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:56.092456 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:56.736903 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:56.736860 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:57.414621 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.414589 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:57.604906 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.604872 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p"] Oct 31 00:44:57.608602 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.608586 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.615073 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.615054 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-47f370-openshift-default-dockercfg-bxkmf\"" Oct 31 00:44:57.622927 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.622898 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p"] Oct 31 00:44:57.653827 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.653787 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffgwf\" (UniqueName: \"kubernetes.io/projected/479718cc-f691-42a3-b6be-8f12366835cc-kube-api-access-ffgwf\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.653996 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.653844 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-workload-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.653996 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.653917 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479718cc-f691-42a3-b6be-8f12366835cc-istio-podinfo\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.654092 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.654012 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-workload-certs\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.654126 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.654101 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479718cc-f691-42a3-b6be-8f12366835cc-istiod-ca-cert\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.654180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.654164 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-istio-envoy\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.654222 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.654208 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479718cc-f691-42a3-b6be-8f12366835cc-istio-token\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.654268 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.654229 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-istio-data\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.654324 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.654309 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-credential-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.741492 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.741462 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9"] Oct 31 00:44:57.745081 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.745062 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.755506 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.755485 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-5c2a03-openshift-default-dockercfg-6267p\"" Oct 31 00:44:57.755584 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.755557 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-istio-envoy\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.755664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.755580 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479718cc-f691-42a3-b6be-8f12366835cc-istio-token\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.755664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.755600 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-istio-data\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.755779 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.755751 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-credential-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.755838 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.755794 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffgwf\" (UniqueName: \"kubernetes.io/projected/479718cc-f691-42a3-b6be-8f12366835cc-kube-api-access-ffgwf\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.755893 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.755838 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-workload-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.755893 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.755865 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479718cc-f691-42a3-b6be-8f12366835cc-istio-podinfo\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.756003 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.755917 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-workload-certs\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.756003 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.755992 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479718cc-f691-42a3-b6be-8f12366835cc-istiod-ca-cert\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.756117 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.756004 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-istio-data\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.756188 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.756159 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-credential-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.756251 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.756221 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-workload-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.757036 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.757015 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-workload-certs\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.757184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.757129 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479718cc-f691-42a3-b6be-8f12366835cc-istiod-ca-cert\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.759008 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.758984 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479718cc-f691-42a3-b6be-8f12366835cc-istio-podinfo\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.760545 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.760523 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9"] Oct 31 00:44:57.762828 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.762783 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-istio-envoy\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.766389 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.766365 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479718cc-f691-42a3-b6be-8f12366835cc-istio-token\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.766552 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.766529 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffgwf\" (UniqueName: \"kubernetes.io/projected/479718cc-f691-42a3-b6be-8f12366835cc-kube-api-access-ffgwf\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.857104 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.857067 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-istio-envoy\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.857273 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.857114 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czbx5\" (UniqueName: \"kubernetes.io/projected/fddc64cf-c911-4921-b176-a7c119cdd118-kube-api-access-czbx5\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.857273 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.857230 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-workload-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.857273 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.857249 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-credential-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.857455 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.857275 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fddc64cf-c911-4921-b176-a7c119cdd118-istio-token\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.857455 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.857299 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fddc64cf-c911-4921-b176-a7c119cdd118-istiod-ca-cert\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.857455 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.857349 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-istio-data\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.857710 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.857557 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fddc64cf-c911-4921-b176-a7c119cdd118-istio-podinfo\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.857710 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.857604 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-workload-certs\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.920615 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.920578 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:57.958993 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.958962 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czbx5\" (UniqueName: \"kubernetes.io/projected/fddc64cf-c911-4921-b176-a7c119cdd118-kube-api-access-czbx5\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.959167 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.959013 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-workload-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.959167 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.959033 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-credential-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.959167 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.959069 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fddc64cf-c911-4921-b176-a7c119cdd118-istio-token\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.959167 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.959098 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fddc64cf-c911-4921-b176-a7c119cdd118-istiod-ca-cert\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.959167 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.959128 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-istio-data\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.959436 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.959175 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fddc64cf-c911-4921-b176-a7c119cdd118-istio-podinfo\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.959436 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.959205 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-workload-certs\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.959436 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.959306 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-istio-envoy\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.959597 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.959467 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-workload-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.959597 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.959511 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-credential-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.959722 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.959704 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-istio-data\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.959867 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.959849 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-workload-certs\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.960132 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.960109 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fddc64cf-c911-4921-b176-a7c119cdd118-istiod-ca-cert\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.962101 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.962054 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fddc64cf-c911-4921-b176-a7c119cdd118-istio-podinfo\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.962101 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.962091 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-istio-envoy\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.975430 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.975403 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fddc64cf-c911-4921-b176-a7c119cdd118-istio-token\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:57.977563 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:57.977535 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czbx5\" (UniqueName: \"kubernetes.io/projected/fddc64cf-c911-4921-b176-a7c119cdd118-kube-api-access-czbx5\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:58.058156 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:58.058080 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:44:58.204552 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:58.204528 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p"] Oct 31 00:44:58.206723 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:44:58.206697 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479718cc_f691_42a3_b6be_8f12366835cc.slice/crio-a371c405e873a6e018a35f45bc41e5c27ec217ec2ec21b31701e79a3f94721e7 WatchSource:0}: Error finding container a371c405e873a6e018a35f45bc41e5c27ec217ec2ec21b31701e79a3f94721e7: Status 404 returned error can't find the container with id a371c405e873a6e018a35f45bc41e5c27ec217ec2ec21b31701e79a3f94721e7 Oct 31 00:44:58.209073 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:58.209043 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:44:58.209141 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:58.209108 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:44:58.209181 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:58.209141 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:44:58.243720 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:58.243698 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:58.345278 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:58.345250 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9"] Oct 31 00:44:58.348383 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:44:58.348346 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfddc64cf_c911_4921_b176_a7c119cdd118.slice/crio-afac849f835049779142947fbc32c79a3654b999968e81217cd31b54a621cf19 WatchSource:0}: Error finding container afac849f835049779142947fbc32c79a3654b999968e81217cd31b54a621cf19: Status 404 returned error can't find the container with id afac849f835049779142947fbc32c79a3654b999968e81217cd31b54a621cf19 Oct 31 00:44:58.351096 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:58.351041 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:44:58.351187 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:58.351138 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:44:58.351227 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:58.351185 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:44:59.007280 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:59.007247 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:59.067659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:59.066868 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" event={"ID":"479718cc-f691-42a3-b6be-8f12366835cc","Type":"ContainerStarted","Data":"33940ca52a3535c1baf63cfe4dd7870ca997b77a4bc1ae54fd366804a0fda590"} Oct 31 00:44:59.067659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:59.066935 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" event={"ID":"479718cc-f691-42a3-b6be-8f12366835cc","Type":"ContainerStarted","Data":"a371c405e873a6e018a35f45bc41e5c27ec217ec2ec21b31701e79a3f94721e7"} Oct 31 00:44:59.074951 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:59.073209 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" event={"ID":"fddc64cf-c911-4921-b176-a7c119cdd118","Type":"ContainerStarted","Data":"9fed18f5c0ad267bdf0f1cd44b9a7f985195622529b61a5fefff5b6c56dccff0"} Oct 31 00:44:59.074951 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:59.073271 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" event={"ID":"fddc64cf-c911-4921-b176-a7c119cdd118","Type":"ContainerStarted","Data":"afac849f835049779142947fbc32c79a3654b999968e81217cd31b54a621cf19"} Oct 31 00:44:59.132924 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:59.132370 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" podStartSLOduration=2.132350513 podStartE2EDuration="2.132350513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:59.105185913 +0000 UTC m=+695.134366315" watchObservedRunningTime="2026-10-31 00:44:59.132350513 +0000 UTC m=+695.161530904" Oct 31 00:44:59.152330 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:59.138298 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" podStartSLOduration=2.138277429 podStartE2EDuration="2.138277429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:59.13398679 +0000 UTC m=+695.163167181" watchObservedRunningTime="2026-10-31 00:44:59.138277429 +0000 UTC m=+695.167457824" Oct 31 00:44:59.772920 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:59.772766 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:59.921185 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:59.921080 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:44:59.922433 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:59.922405 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 00:44:59.922541 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:44:59.922458 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" podUID="479718cc-f691-42a3-b6be-8f12366835cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 00:45:00.064718 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.063752 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:45:00.064718 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.063927 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 00:45:00.064718 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.063978 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" podUID="fddc64cf-c911-4921-b176-a7c119cdd118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 00:45:00.173522 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.173489 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl"] Oct 31 00:45:00.180364 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.178131 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d"] Oct 31 00:45:00.180364 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.179945 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" Oct 31 00:45:00.185228 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.185196 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d"] Oct 31 00:45:00.185389 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.185248 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl"] Oct 31 00:45:00.185389 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.185350 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" Oct 31 00:45:00.186190 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.185751 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24a1307-b598-4f8f-860c-d59bb860c3c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl\" (UID: \"a24a1307-b598-4f8f-860c-d59bb860c3c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" Oct 31 00:45:00.186190 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.185829 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6vj4\" (UniqueName: \"kubernetes.io/projected/a24a1307-b598-4f8f-860c-d59bb860c3c7-kube-api-access-j6vj4\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl\" (UID: \"a24a1307-b598-4f8f-860c-d59bb860c3c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" Oct 31 00:45:00.192030 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.191995 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w"] Oct 31 00:45:00.195774 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.195751 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" Oct 31 00:45:00.204201 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.204179 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w"] Oct 31 00:45:00.281100 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.281067 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:45:00.286989 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.286647 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84dc7f22-aca3-4a19-9805-1b5c8fa92863-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d\" (UID: \"84dc7f22-aca3-4a19-9805-1b5c8fa92863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" Oct 31 00:45:00.286989 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.286704 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdqsj\" (UniqueName: \"kubernetes.io/projected/84dc7f22-aca3-4a19-9805-1b5c8fa92863-kube-api-access-bdqsj\") pod \"maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d\" (UID: \"84dc7f22-aca3-4a19-9805-1b5c8fa92863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" Oct 31 00:45:00.286989 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.286737 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24a1307-b598-4f8f-860c-d59bb860c3c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl\" (UID: \"a24a1307-b598-4f8f-860c-d59bb860c3c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" Oct 31 00:45:00.287575 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.287536 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24a1307-b598-4f8f-860c-d59bb860c3c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl\" (UID: \"a24a1307-b598-4f8f-860c-d59bb860c3c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" Oct 31 00:45:00.287734 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.287652 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krd69\" (UniqueName: \"kubernetes.io/projected/f8a53a21-ed7f-4c46-aad3-4b9c68733d01-kube-api-access-krd69\") pod \"maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w\" (UID: \"f8a53a21-ed7f-4c46-aad3-4b9c68733d01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" Oct 31 00:45:00.287734 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.287694 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6vj4\" (UniqueName: \"kubernetes.io/projected/a24a1307-b598-4f8f-860c-d59bb860c3c7-kube-api-access-j6vj4\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl\" (UID: \"a24a1307-b598-4f8f-860c-d59bb860c3c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" Oct 31 00:45:00.287814 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.287721 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a53a21-ed7f-4c46-aad3-4b9c68733d01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w\" (UID: \"f8a53a21-ed7f-4c46-aad3-4b9c68733d01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" Oct 31 00:45:00.297203 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.297173 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6vj4\" (UniqueName: \"kubernetes.io/projected/a24a1307-b598-4f8f-860c-d59bb860c3c7-kube-api-access-j6vj4\") pod \"maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl\" (UID: \"a24a1307-b598-4f8f-860c-d59bb860c3c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" Oct 31 00:45:00.389021 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.388929 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krd69\" (UniqueName: \"kubernetes.io/projected/f8a53a21-ed7f-4c46-aad3-4b9c68733d01-kube-api-access-krd69\") pod \"maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w\" (UID: \"f8a53a21-ed7f-4c46-aad3-4b9c68733d01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" Oct 31 00:45:00.389021 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.388977 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a53a21-ed7f-4c46-aad3-4b9c68733d01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w\" (UID: \"f8a53a21-ed7f-4c46-aad3-4b9c68733d01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" Oct 31 00:45:00.389285 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.389060 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84dc7f22-aca3-4a19-9805-1b5c8fa92863-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d\" (UID: \"84dc7f22-aca3-4a19-9805-1b5c8fa92863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" Oct 31 00:45:00.389285 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.389090 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdqsj\" (UniqueName: \"kubernetes.io/projected/84dc7f22-aca3-4a19-9805-1b5c8fa92863-kube-api-access-bdqsj\") pod \"maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d\" (UID: \"84dc7f22-aca3-4a19-9805-1b5c8fa92863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" Oct 31 00:45:00.389767 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.389744 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a53a21-ed7f-4c46-aad3-4b9c68733d01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w\" (UID: \"f8a53a21-ed7f-4c46-aad3-4b9c68733d01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" Oct 31 00:45:00.389823 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.389807 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84dc7f22-aca3-4a19-9805-1b5c8fa92863-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d\" (UID: \"84dc7f22-aca3-4a19-9805-1b5c8fa92863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" Oct 31 00:45:00.399813 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.399778 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdqsj\" (UniqueName: \"kubernetes.io/projected/84dc7f22-aca3-4a19-9805-1b5c8fa92863-kube-api-access-bdqsj\") pod \"maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d\" (UID: \"84dc7f22-aca3-4a19-9805-1b5c8fa92863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" Oct 31 00:45:00.399984 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.399968 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krd69\" (UniqueName: \"kubernetes.io/projected/f8a53a21-ed7f-4c46-aad3-4b9c68733d01-kube-api-access-krd69\") pod \"maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w\" (UID: \"f8a53a21-ed7f-4c46-aad3-4b9c68733d01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" Oct 31 00:45:00.501282 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.501245 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" Oct 31 00:45:00.513789 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.513768 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" Oct 31 00:45:00.517883 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.517861 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" Oct 31 00:45:00.669789 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.669594 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:00.909974 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.847597 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl"] Oct 31 00:45:00.938129 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.929713 4519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:45:00.947647 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.946265 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 00:45:00.947647 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.946323 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" podUID="479718cc-f691-42a3-b6be-8f12366835cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 00:45:00.947647 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:45:00.946724 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24a1307_b598_4f8f_860c_d59bb860c3c7.slice/crio-c6ee5288d1245a43034e5331a8347193a569ae96722532541eec6974d964ee21 WatchSource:0}: Error finding container c6ee5288d1245a43034e5331a8347193a569ae96722532541eec6974d964ee21: Status 404 returned error can't find the container with id c6ee5288d1245a43034e5331a8347193a569ae96722532541eec6974d964ee21 Oct 31 00:45:00.987762 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:45:00.987719 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84dc7f22_aca3_4a19_9805_1b5c8fa92863.slice/crio-039683fa9dae906646b9433550e15545f667c351b76daee9200dd0c20ac46819 WatchSource:0}: Error finding container 039683fa9dae906646b9433550e15545f667c351b76daee9200dd0c20ac46819: Status 404 returned error can't find the container with id 039683fa9dae906646b9433550e15545f667c351b76daee9200dd0c20ac46819 Oct 31 00:45:00.996949 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.996867 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d"] Oct 31 00:45:00.998378 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:00.997832 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w"] Oct 31 00:45:01.073805 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:01.073772 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 00:45:01.074233 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:01.074209 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" podUID="fddc64cf-c911-4921-b176-a7c119cdd118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 00:45:01.086691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:01.086661 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" event={"ID":"84dc7f22-aca3-4a19-9805-1b5c8fa92863","Type":"ContainerStarted","Data":"039683fa9dae906646b9433550e15545f667c351b76daee9200dd0c20ac46819"} Oct 31 00:45:01.088123 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:01.088083 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" event={"ID":"f8a53a21-ed7f-4c46-aad3-4b9c68733d01","Type":"ContainerStarted","Data":"0d5990949eadbfaad209801a91d225b5385e256883e6a5f317ceed6ac211055f"} Oct 31 00:45:01.089780 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:01.089761 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" event={"ID":"a24a1307-b598-4f8f-860c-d59bb860c3c7","Type":"ContainerStarted","Data":"c6ee5288d1245a43034e5331a8347193a569ae96722532541eec6974d964ee21"} Oct 31 00:45:01.703302 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:01.703265 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:01.921066 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:01.921028 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 00:45:01.921247 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:01.921102 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" podUID="479718cc-f691-42a3-b6be-8f12366835cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 00:45:02.059080 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:02.059041 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 00:45:02.059346 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:02.059322 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" podUID="fddc64cf-c911-4921-b176-a7c119cdd118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 00:45:02.576184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:02.576107 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:02.922239 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:02.921761 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 00:45:02.922239 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:02.921836 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" podUID="479718cc-f691-42a3-b6be-8f12366835cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 00:45:03.064209 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:03.063770 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 00:45:03.064209 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:03.063830 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" podUID="fddc64cf-c911-4921-b176-a7c119cdd118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 00:45:03.465659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:03.464964 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:03.921242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:03.921209 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 00:45:03.921757 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:03.921272 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" podUID="479718cc-f691-42a3-b6be-8f12366835cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 00:45:04.071777 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:04.071318 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 00:45:04.071777 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:04.071379 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" podUID="fddc64cf-c911-4921-b176-a7c119cdd118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 00:45:04.177758 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:04.177624 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:04.787066 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:04.787021 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59"] Oct 31 00:45:04.800982 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:04.800947 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59"] Oct 31 00:45:04.801158 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:04.801133 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" Oct 31 00:45:04.808840 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:04.808797 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:45:04.843096 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:04.843056 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:04.897228 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:04.897185 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324ec116-8c6f-4983-b1eb-99f935102520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-310a80d5-tjk59\" (UID: \"324ec116-8c6f-4983-b1eb-99f935102520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" Oct 31 00:45:04.897407 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:04.897308 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324ec116-8c6f-4983-b1eb-99f935102520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-310a80d5-tjk59\" (UID: \"324ec116-8c6f-4983-b1eb-99f935102520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" Oct 31 00:45:04.897407 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:04.897365 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz7l2\" (UniqueName: \"kubernetes.io/projected/324ec116-8c6f-4983-b1eb-99f935102520-kube-api-access-kz7l2\") pod \"maas-api-revoke-sub-310a80d5-tjk59\" (UID: \"324ec116-8c6f-4983-b1eb-99f935102520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" Oct 31 00:45:04.921868 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:04.921227 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 00:45:04.921868 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:04.921301 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" podUID="479718cc-f691-42a3-b6be-8f12366835cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 00:45:04.998937 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:04.998293 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324ec116-8c6f-4983-b1eb-99f935102520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-310a80d5-tjk59\" (UID: \"324ec116-8c6f-4983-b1eb-99f935102520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" Oct 31 00:45:04.998937 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:04.998379 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz7l2\" (UniqueName: \"kubernetes.io/projected/324ec116-8c6f-4983-b1eb-99f935102520-kube-api-access-kz7l2\") pod \"maas-api-revoke-sub-310a80d5-tjk59\" (UID: \"324ec116-8c6f-4983-b1eb-99f935102520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" Oct 31 00:45:04.998937 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:04.998454 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324ec116-8c6f-4983-b1eb-99f935102520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-310a80d5-tjk59\" (UID: \"324ec116-8c6f-4983-b1eb-99f935102520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" Oct 31 00:45:04.999189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:04.999158 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324ec116-8c6f-4983-b1eb-99f935102520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-310a80d5-tjk59\" (UID: \"324ec116-8c6f-4983-b1eb-99f935102520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" Oct 31 00:45:05.005046 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:05.005013 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324ec116-8c6f-4983-b1eb-99f935102520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-310a80d5-tjk59\" (UID: \"324ec116-8c6f-4983-b1eb-99f935102520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" Oct 31 00:45:05.010299 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:05.010265 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz7l2\" (UniqueName: \"kubernetes.io/projected/324ec116-8c6f-4983-b1eb-99f935102520-kube-api-access-kz7l2\") pod \"maas-api-revoke-sub-310a80d5-tjk59\" (UID: \"324ec116-8c6f-4983-b1eb-99f935102520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" Oct 31 00:45:05.059618 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:05.059128 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 00:45:05.059618 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:05.059194 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" podUID="fddc64cf-c911-4921-b176-a7c119cdd118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 00:45:05.135168 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:05.135130 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" Oct 31 00:45:05.292298 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:05.292155 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:05.464795 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:45:05.464764 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324ec116_8c6f_4983_b1eb_99f935102520.slice/crio-8e229a07926e8af9dbd4dcc35e6a3b21fe38a75b8a0872c609a4993026f21ecc WatchSource:0}: Error finding container 8e229a07926e8af9dbd4dcc35e6a3b21fe38a75b8a0872c609a4993026f21ecc: Status 404 returned error can't find the container with id 8e229a07926e8af9dbd4dcc35e6a3b21fe38a75b8a0872c609a4993026f21ecc Oct 31 00:45:05.467832 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:05.467694 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59"] Oct 31 00:45:05.648748 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:05.648711 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:05.927741 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:05.927708 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:45:05.928194 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:05.928161 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:45:05.928939 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:05.928915 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:45:06.064118 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:06.064087 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:45:06.065169 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:06.065141 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:45:06.065698 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:06.065674 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:45:06.141606 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:06.141539 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" event={"ID":"324ec116-8c6f-4983-b1eb-99f935102520","Type":"ContainerStarted","Data":"8e229a07926e8af9dbd4dcc35e6a3b21fe38a75b8a0872c609a4993026f21ecc"} Oct 31 00:45:06.368549 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:06.368518 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:07.036436 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:07.036405 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:07.147519 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:07.147485 4519 generic.go:356] "Generic (PLEG): container finished" podID="84dc7f22-aca3-4a19-9805-1b5c8fa92863" containerID="fb13a4afe062934ed4212f07ddcf18f3733fec69642e1af972cefdfddde759b3" exitCode=60 Oct 31 00:45:07.147739 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:07.147572 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" event={"ID":"84dc7f22-aca3-4a19-9805-1b5c8fa92863","Type":"ContainerDied","Data":"fb13a4afe062934ed4212f07ddcf18f3733fec69642e1af972cefdfddde759b3"} Oct 31 00:45:07.147869 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:07.147851 4519 scope.go:122] "RemoveContainer" containerID="fb13a4afe062934ed4212f07ddcf18f3733fec69642e1af972cefdfddde759b3" Oct 31 00:45:07.149155 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:07.149131 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" event={"ID":"324ec116-8c6f-4983-b1eb-99f935102520","Type":"ContainerStarted","Data":"0daf1901252d23296a1526169e67a9eb2a0d0eb1c935d0d1bc7dc81bd9d53198"} Oct 31 00:45:07.150932 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:07.150889 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" event={"ID":"f8a53a21-ed7f-4c46-aad3-4b9c68733d01","Type":"ContainerStarted","Data":"0bf4ee22fdb08b5e15f03c158fc47405326936e889f80f03974ca86ce21803fc"} Oct 31 00:45:07.153382 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:07.153361 4519 generic.go:356] "Generic (PLEG): container finished" podID="a24a1307-b598-4f8f-860c-d59bb860c3c7" containerID="b467cd0ea436d918688720dd581ba46279ee7c2257f927f64b3d51022f70e673" exitCode=60 Oct 31 00:45:07.153467 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:07.153449 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" event={"ID":"a24a1307-b598-4f8f-860c-d59bb860c3c7","Type":"ContainerDied","Data":"b467cd0ea436d918688720dd581ba46279ee7c2257f927f64b3d51022f70e673"} Oct 31 00:45:07.153598 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:07.153584 4519 scope.go:122] "RemoveContainer" containerID="b467cd0ea436d918688720dd581ba46279ee7c2257f927f64b3d51022f70e673" Oct 31 00:45:07.201317 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:07.201262 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" podStartSLOduration=2.313724353 podStartE2EDuration="3.201243852s" totalImagesPullingTime="887.519499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:07.200392659 +0000 UTC m=+703.229573050" watchObservedRunningTime="2026-10-31 00:45:07.201243852 +0000 UTC m=+703.230424244" Oct 31 00:45:07.218490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:07.218433 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" podStartSLOduration=1.927743352 podStartE2EDuration="7.218419079s" totalImagesPullingTime="5.290675727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:07.215953692 +0000 UTC m=+703.245134081" watchObservedRunningTime="2026-10-31 00:45:07.218419079 +0000 UTC m=+703.247599469" Oct 31 00:45:07.754104 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:07.754068 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:08.159993 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:08.159957 4519 generic.go:356] "Generic (PLEG): container finished" podID="84dc7f22-aca3-4a19-9805-1b5c8fa92863" containerID="105545f29b726128c7d2c8713ca5f286f509a849eca8f4adcaf81668d2eb7ceb" exitCode=60 Oct 31 00:45:08.160465 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:08.160039 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" event={"ID":"84dc7f22-aca3-4a19-9805-1b5c8fa92863","Type":"ContainerDied","Data":"105545f29b726128c7d2c8713ca5f286f509a849eca8f4adcaf81668d2eb7ceb"} Oct 31 00:45:08.160465 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:08.160099 4519 scope.go:122] "RemoveContainer" containerID="fb13a4afe062934ed4212f07ddcf18f3733fec69642e1af972cefdfddde759b3" Oct 31 00:45:08.160465 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:08.160400 4519 scope.go:122] "RemoveContainer" containerID="105545f29b726128c7d2c8713ca5f286f509a849eca8f4adcaf81668d2eb7ceb" Oct 31 00:45:08.160706 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:45:08.160685 4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d_odh-ai-gateway-infra(84dc7f22-aca3-4a19-9805-1b5c8fa92863)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" podUID="84dc7f22-aca3-4a19-9805-1b5c8fa92863" Oct 31 00:45:08.161583 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:08.161560 4519 generic.go:356] "Generic (PLEG): container finished" podID="324ec116-8c6f-4983-b1eb-99f935102520" containerID="0daf1901252d23296a1526169e67a9eb2a0d0eb1c935d0d1bc7dc81bd9d53198" exitCode=0 Oct 31 00:45:08.161694 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:08.161654 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" event={"ID":"324ec116-8c6f-4983-b1eb-99f935102520","Type":"ContainerDied","Data":"0daf1901252d23296a1526169e67a9eb2a0d0eb1c935d0d1bc7dc81bd9d53198"} Oct 31 00:45:08.163205 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:08.163182 4519 generic.go:356] "Generic (PLEG): container finished" podID="f8a53a21-ed7f-4c46-aad3-4b9c68733d01" containerID="0bf4ee22fdb08b5e15f03c158fc47405326936e889f80f03974ca86ce21803fc" exitCode=60 Oct 31 00:45:08.163301 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:08.163248 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" event={"ID":"f8a53a21-ed7f-4c46-aad3-4b9c68733d01","Type":"ContainerDied","Data":"0bf4ee22fdb08b5e15f03c158fc47405326936e889f80f03974ca86ce21803fc"} Oct 31 00:45:08.163459 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:08.163443 4519 scope.go:122] "RemoveContainer" containerID="0bf4ee22fdb08b5e15f03c158fc47405326936e889f80f03974ca86ce21803fc" Oct 31 00:45:08.165108 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:08.165084 4519 generic.go:356] "Generic (PLEG): container finished" podID="a24a1307-b598-4f8f-860c-d59bb860c3c7" containerID="aba6c40f302a8e4262b0c169bdce8f5ae909d411a2474ab5fbeba0da9750a593" exitCode=60 Oct 31 00:45:08.165196 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:08.165127 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" event={"ID":"a24a1307-b598-4f8f-860c-d59bb860c3c7","Type":"ContainerDied","Data":"aba6c40f302a8e4262b0c169bdce8f5ae909d411a2474ab5fbeba0da9750a593"} Oct 31 00:45:08.165421 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:08.165403 4519 scope.go:122] "RemoveContainer" containerID="aba6c40f302a8e4262b0c169bdce8f5ae909d411a2474ab5fbeba0da9750a593" Oct 31 00:45:08.165690 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:45:08.165669 4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl_odh-ai-gateway-infra(a24a1307-b598-4f8f-860c-d59bb860c3c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" podUID="a24a1307-b598-4f8f-860c-d59bb860c3c7" Oct 31 00:45:08.180468 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:08.180444 4519 scope.go:122] "RemoveContainer" containerID="b467cd0ea436d918688720dd581ba46279ee7c2257f927f64b3d51022f70e673" Oct 31 00:45:08.485577 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:08.485543 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:09.171287 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.171259 4519 scope.go:122] "RemoveContainer" containerID="105545f29b726128c7d2c8713ca5f286f509a849eca8f4adcaf81668d2eb7ceb" Oct 31 00:45:09.171740 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:45:09.171519 4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d_odh-ai-gateway-infra(84dc7f22-aca3-4a19-9805-1b5c8fa92863)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" podUID="84dc7f22-aca3-4a19-9805-1b5c8fa92863" Oct 31 00:45:09.173082 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.173019 4519 generic.go:356] "Generic (PLEG): container finished" podID="f8a53a21-ed7f-4c46-aad3-4b9c68733d01" containerID="c2ce7b2fda52e4e1b293d8e06068e22aab5473f33a90d286ffc43015cff1c460" exitCode=60 Oct 31 00:45:09.173192 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.173102 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" event={"ID":"f8a53a21-ed7f-4c46-aad3-4b9c68733d01","Type":"ContainerDied","Data":"c2ce7b2fda52e4e1b293d8e06068e22aab5473f33a90d286ffc43015cff1c460"} Oct 31 00:45:09.173192 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.173169 4519 scope.go:122] "RemoveContainer" containerID="0bf4ee22fdb08b5e15f03c158fc47405326936e889f80f03974ca86ce21803fc" Oct 31 00:45:09.173327 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.173308 4519 scope.go:122] "RemoveContainer" containerID="c2ce7b2fda52e4e1b293d8e06068e22aab5473f33a90d286ffc43015cff1c460" Oct 31 00:45:09.173575 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:45:09.173552 4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w_odh-ai-gateway-infra(f8a53a21-ed7f-4c46-aad3-4b9c68733d01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" podUID="f8a53a21-ed7f-4c46-aad3-4b9c68733d01" Oct 31 00:45:09.174916 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.174894 4519 scope.go:122] "RemoveContainer" containerID="aba6c40f302a8e4262b0c169bdce8f5ae909d411a2474ab5fbeba0da9750a593" Oct 31 00:45:09.175090 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:45:09.175078 4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl_odh-ai-gateway-infra(a24a1307-b598-4f8f-860c-d59bb860c3c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" podUID="a24a1307-b598-4f8f-860c-d59bb860c3c7" Oct 31 00:45:09.248670 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.242155 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:09.331954 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.331919 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" Oct 31 00:45:09.377057 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.375943 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324ec116-8c6f-4983-b1eb-99f935102520-kube-api-access-kz7l2\" (UniqueName: \"kubernetes.io/projected/324ec116-8c6f-4983-b1eb-99f935102520-kube-api-access-kz7l2\") pod \"324ec116-8c6f-4983-b1eb-99f935102520\" (UID: \"324ec116-8c6f-4983-b1eb-99f935102520\") " Oct 31 00:45:09.377057 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.376110 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/324ec116-8c6f-4983-b1eb-99f935102520-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324ec116-8c6f-4983-b1eb-99f935102520-maas-api-openshift-ca\") pod \"324ec116-8c6f-4983-b1eb-99f935102520\" (UID: \"324ec116-8c6f-4983-b1eb-99f935102520\") " Oct 31 00:45:09.377057 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.376173 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/324ec116-8c6f-4983-b1eb-99f935102520-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324ec116-8c6f-4983-b1eb-99f935102520-maas-api-xks-ca\") pod \"324ec116-8c6f-4983-b1eb-99f935102520\" (UID: \"324ec116-8c6f-4983-b1eb-99f935102520\") " Oct 31 00:45:09.379079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.379041 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324ec116-8c6f-4983-b1eb-99f935102520-maas-api-openshift-ca" pod "324ec116-8c6f-4983-b1eb-99f935102520" (UID: "324ec116-8c6f-4983-b1eb-99f935102520"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:09.382418 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.382385 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324ec116-8c6f-4983-b1eb-99f935102520-kube-api-access-kz7l2" pod "324ec116-8c6f-4983-b1eb-99f935102520" (UID: "324ec116-8c6f-4983-b1eb-99f935102520"). InnerVolumeSpecName "kube-api-access-kz7l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:09.383251 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.383225 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/324ec116-8c6f-4983-b1eb-99f935102520-maas-api-xks-ca" pod "324ec116-8c6f-4983-b1eb-99f935102520" (UID: "324ec116-8c6f-4983-b1eb-99f935102520"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:09.477489 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.477403 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324ec116-8c6f-4983-b1eb-99f935102520-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:09.477489 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.477435 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324ec116-8c6f-4983-b1eb-99f935102520-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:09.477489 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.477447 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz7l2\" (UniqueName: \"kubernetes.io/projected/324ec116-8c6f-4983-b1eb-99f935102520-kube-api-access-kz7l2\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:09.849820 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:09.849781 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:10.181458 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:10.181428 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" Oct 31 00:45:10.181962 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:10.181475 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59" event={"ID":"324ec116-8c6f-4983-b1eb-99f935102520","Type":"ContainerDied","Data":"8e229a07926e8af9dbd4dcc35e6a3b21fe38a75b8a0872c609a4993026f21ecc"} Oct 31 00:45:10.181962 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:10.181521 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e229a07926e8af9dbd4dcc35e6a3b21fe38a75b8a0872c609a4993026f21ecc" Oct 31 00:45:10.183598 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:10.183573 4519 scope.go:122] "RemoveContainer" containerID="c2ce7b2fda52e4e1b293d8e06068e22aab5473f33a90d286ffc43015cff1c460" Oct 31 00:45:10.183904 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:45:10.183883 4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w_odh-ai-gateway-infra(f8a53a21-ed7f-4c46-aad3-4b9c68733d01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" podUID="f8a53a21-ed7f-4c46-aad3-4b9c68733d01" Oct 31 00:45:10.341433 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:10.341404 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:10.530132 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:10.530096 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:11.315225 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:11.315188 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:12.338776 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:12.338734 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:13.032682 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:13.032616 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:13.638016 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:13.637979 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:13.738994 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:13.738961 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:14.354185 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:14.354133 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:15.175785 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.175748 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:15.409463 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.409420 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49"] Oct 31 00:45:15.410041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.410023 4519 cpu_manager.go:414] "containerMap: removing container" podUID="324ec116-8c6f-4983-b1eb-99f935102520" containerName="revoke-keys" Oct 31 00:45:15.410041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.410042 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324ec116-8c6f-4983-b1eb-99f935102520" containerName="revoke-keys" Oct 31 00:45:15.410175 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.410109 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="324ec116-8c6f-4983-b1eb-99f935102520" containerName="revoke-keys" Oct 31 00:45:15.413485 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.413464 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49" Oct 31 00:45:15.418338 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.418312 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:45:15.427283 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.425515 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49"] Oct 31 00:45:15.453399 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.453347 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76xwg\" (UniqueName: \"kubernetes.io/projected/6dfb7609-1a91-4376-90d1-b4baf6047398-kube-api-access-76xwg\") pod \"maas-api-revoke-sub-909ea630-66j49\" (UID: \"6dfb7609-1a91-4376-90d1-b4baf6047398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49" Oct 31 00:45:15.453571 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.453463 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dfb7609-1a91-4376-90d1-b4baf6047398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-909ea630-66j49\" (UID: \"6dfb7609-1a91-4376-90d1-b4baf6047398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49" Oct 31 00:45:15.453571 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.453512 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dfb7609-1a91-4376-90d1-b4baf6047398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-909ea630-66j49\" (UID: \"6dfb7609-1a91-4376-90d1-b4baf6047398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49" Oct 31 00:45:15.555285 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.555206 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dfb7609-1a91-4376-90d1-b4baf6047398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-909ea630-66j49\" (UID: \"6dfb7609-1a91-4376-90d1-b4baf6047398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49" Oct 31 00:45:15.555285 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.555277 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dfb7609-1a91-4376-90d1-b4baf6047398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-909ea630-66j49\" (UID: \"6dfb7609-1a91-4376-90d1-b4baf6047398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49" Oct 31 00:45:15.555679 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.555438 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76xwg\" (UniqueName: \"kubernetes.io/projected/6dfb7609-1a91-4376-90d1-b4baf6047398-kube-api-access-76xwg\") pod \"maas-api-revoke-sub-909ea630-66j49\" (UID: \"6dfb7609-1a91-4376-90d1-b4baf6047398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49" Oct 31 00:45:15.556042 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.556010 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dfb7609-1a91-4376-90d1-b4baf6047398-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-909ea630-66j49\" (UID: \"6dfb7609-1a91-4376-90d1-b4baf6047398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49" Oct 31 00:45:15.557800 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.557781 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dfb7609-1a91-4376-90d1-b4baf6047398-maas-api-xks-ca\") pod \"maas-api-revoke-sub-909ea630-66j49\" (UID: \"6dfb7609-1a91-4376-90d1-b4baf6047398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49" Oct 31 00:45:15.564904 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.564881 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76xwg\" (UniqueName: \"kubernetes.io/projected/6dfb7609-1a91-4376-90d1-b4baf6047398-kube-api-access-76xwg\") pod \"maas-api-revoke-sub-909ea630-66j49\" (UID: \"6dfb7609-1a91-4376-90d1-b4baf6047398\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49" Oct 31 00:45:15.740029 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.739998 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49" Oct 31 00:45:15.931983 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:15.931943 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:16.038727 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:16.038699 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49"] Oct 31 00:45:16.040269 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:45:16.040241 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dfb7609_1a91_4376_90d1_b4baf6047398.slice/crio-ed15ec42c5ae4a4d51ea8a231e8ec3af96fbcd2e5a3cc01cd0b3f89c7c635a27 WatchSource:0}: Error finding container ed15ec42c5ae4a4d51ea8a231e8ec3af96fbcd2e5a3cc01cd0b3f89c7c635a27: Status 404 returned error can't find the container with id ed15ec42c5ae4a4d51ea8a231e8ec3af96fbcd2e5a3cc01cd0b3f89c7c635a27 Oct 31 00:45:16.213517 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:16.213484 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49" event={"ID":"6dfb7609-1a91-4376-90d1-b4baf6047398","Type":"ContainerStarted","Data":"ed15ec42c5ae4a4d51ea8a231e8ec3af96fbcd2e5a3cc01cd0b3f89c7c635a27"} Oct 31 00:45:16.587945 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:16.587869 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp"] Oct 31 00:45:16.591983 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:16.591962 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp" Oct 31 00:45:16.600496 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:16.600443 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp"] Oct 31 00:45:16.647406 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:16.647375 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:16.666692 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:16.666658 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c603a9a-871a-40bc-86e7-022941340d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-874cf9d5-kn2rp\" (UID: \"0c603a9a-871a-40bc-86e7-022941340d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp" Oct 31 00:45:16.666692 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:16.666695 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c603a9a-871a-40bc-86e7-022941340d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-874cf9d5-kn2rp\" (UID: \"0c603a9a-871a-40bc-86e7-022941340d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp" Oct 31 00:45:16.666897 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:16.666835 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6sl4\" (UniqueName: \"kubernetes.io/projected/0c603a9a-871a-40bc-86e7-022941340d89-kube-api-access-d6sl4\") pod \"maas-api-revoke-sub-874cf9d5-kn2rp\" (UID: \"0c603a9a-871a-40bc-86e7-022941340d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp" Oct 31 00:45:16.767469 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:16.767431 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c603a9a-871a-40bc-86e7-022941340d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-874cf9d5-kn2rp\" (UID: \"0c603a9a-871a-40bc-86e7-022941340d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp" Oct 31 00:45:16.767469 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:16.767467 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c603a9a-871a-40bc-86e7-022941340d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-874cf9d5-kn2rp\" (UID: \"0c603a9a-871a-40bc-86e7-022941340d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp" Oct 31 00:45:16.767763 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:16.767524 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6sl4\" (UniqueName: \"kubernetes.io/projected/0c603a9a-871a-40bc-86e7-022941340d89-kube-api-access-d6sl4\") pod \"maas-api-revoke-sub-874cf9d5-kn2rp\" (UID: \"0c603a9a-871a-40bc-86e7-022941340d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp" Oct 31 00:45:16.768124 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:16.768101 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c603a9a-871a-40bc-86e7-022941340d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-874cf9d5-kn2rp\" (UID: \"0c603a9a-871a-40bc-86e7-022941340d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp" Oct 31 00:45:16.769956 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:16.769932 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c603a9a-871a-40bc-86e7-022941340d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-874cf9d5-kn2rp\" (UID: \"0c603a9a-871a-40bc-86e7-022941340d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp" Oct 31 00:45:16.776057 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:16.776037 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6sl4\" (UniqueName: \"kubernetes.io/projected/0c603a9a-871a-40bc-86e7-022941340d89-kube-api-access-d6sl4\") pod \"maas-api-revoke-sub-874cf9d5-kn2rp\" (UID: \"0c603a9a-871a-40bc-86e7-022941340d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp" Oct 31 00:45:16.908179 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:16.908151 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp" Oct 31 00:45:17.185250 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:17.185221 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp"] Oct 31 00:45:17.186454 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:45:17.186428 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c603a9a_871a_40bc_86e7_022941340d89.slice/crio-93542ff898414d6795eaf5f2bf4452da8a1b08a34c4add3f58e6548f6559d301 WatchSource:0}: Error finding container 93542ff898414d6795eaf5f2bf4452da8a1b08a34c4add3f58e6548f6559d301: Status 404 returned error can't find the container with id 93542ff898414d6795eaf5f2bf4452da8a1b08a34c4add3f58e6548f6559d301 Oct 31 00:45:17.219821 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:17.219790 4519 generic.go:356] "Generic (PLEG): container finished" podID="6dfb7609-1a91-4376-90d1-b4baf6047398" containerID="f4bbad515880662dfeda4693cf930dc7cc85e86dd86e1b5cf47d161202644e75" exitCode=0 Oct 31 00:45:17.220147 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:17.219869 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49" event={"ID":"6dfb7609-1a91-4376-90d1-b4baf6047398","Type":"ContainerDied","Data":"f4bbad515880662dfeda4693cf930dc7cc85e86dd86e1b5cf47d161202644e75"} Oct 31 00:45:17.221037 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:17.221018 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp" event={"ID":"0c603a9a-871a-40bc-86e7-022941340d89","Type":"ContainerStarted","Data":"93542ff898414d6795eaf5f2bf4452da8a1b08a34c4add3f58e6548f6559d301"} Oct 31 00:45:17.357690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:17.357661 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:17.544952 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:45:17.544920 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:45:18.007890 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:18.007858 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:18.227805 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:18.227775 4519 generic.go:356] "Generic (PLEG): container finished" podID="0c603a9a-871a-40bc-86e7-022941340d89" containerID="9ae13c5926dd2434c170467123cb87054517b69d82dcbae50b7c3bcd97e207a1" exitCode=0 Oct 31 00:45:18.228202 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:18.227857 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp" event={"ID":"0c603a9a-871a-40bc-86e7-022941340d89","Type":"ContainerDied","Data":"9ae13c5926dd2434c170467123cb87054517b69d82dcbae50b7c3bcd97e207a1"} Oct 31 00:45:18.377330 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:18.377305 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49" Oct 31 00:45:18.485068 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:18.485034 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dfb7609-1a91-4376-90d1-b4baf6047398-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dfb7609-1a91-4376-90d1-b4baf6047398-maas-api-xks-ca\") pod \"6dfb7609-1a91-4376-90d1-b4baf6047398\" (UID: \"6dfb7609-1a91-4376-90d1-b4baf6047398\") " Oct 31 00:45:18.485220 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:18.485121 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dfb7609-1a91-4376-90d1-b4baf6047398-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dfb7609-1a91-4376-90d1-b4baf6047398-maas-api-openshift-ca\") pod \"6dfb7609-1a91-4376-90d1-b4baf6047398\" (UID: \"6dfb7609-1a91-4376-90d1-b4baf6047398\") " Oct 31 00:45:18.485256 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:18.485225 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dfb7609-1a91-4376-90d1-b4baf6047398-kube-api-access-76xwg\" (UniqueName: \"kubernetes.io/projected/6dfb7609-1a91-4376-90d1-b4baf6047398-kube-api-access-76xwg\") pod \"6dfb7609-1a91-4376-90d1-b4baf6047398\" (UID: \"6dfb7609-1a91-4376-90d1-b4baf6047398\") " Oct 31 00:45:18.485481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:18.485458 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dfb7609-1a91-4376-90d1-b4baf6047398-maas-api-openshift-ca" pod "6dfb7609-1a91-4376-90d1-b4baf6047398" (UID: "6dfb7609-1a91-4376-90d1-b4baf6047398"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:18.485683 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:18.485667 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dfb7609-1a91-4376-90d1-b4baf6047398-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.487252 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:18.487231 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dfb7609-1a91-4376-90d1-b4baf6047398-maas-api-xks-ca" pod "6dfb7609-1a91-4376-90d1-b4baf6047398" (UID: "6dfb7609-1a91-4376-90d1-b4baf6047398"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:18.487317 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:18.487299 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dfb7609-1a91-4376-90d1-b4baf6047398-kube-api-access-76xwg" pod "6dfb7609-1a91-4376-90d1-b4baf6047398" (UID: "6dfb7609-1a91-4376-90d1-b4baf6047398"). InnerVolumeSpecName "kube-api-access-76xwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:18.587079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:18.586997 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76xwg\" (UniqueName: \"kubernetes.io/projected/6dfb7609-1a91-4376-90d1-b4baf6047398-kube-api-access-76xwg\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.587079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:18.587030 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dfb7609-1a91-4376-90d1-b4baf6047398-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.841372 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:18.841283 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:19.233834 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:19.233796 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49" event={"ID":"6dfb7609-1a91-4376-90d1-b4baf6047398","Type":"ContainerDied","Data":"ed15ec42c5ae4a4d51ea8a231e8ec3af96fbcd2e5a3cc01cd0b3f89c7c635a27"} Oct 31 00:45:19.234259 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:19.233864 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed15ec42c5ae4a4d51ea8a231e8ec3af96fbcd2e5a3cc01cd0b3f89c7c635a27" Oct 31 00:45:19.234259 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:19.233812 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49" Oct 31 00:45:19.386168 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:19.386144 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp" Oct 31 00:45:19.496573 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:19.496475 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c603a9a-871a-40bc-86e7-022941340d89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c603a9a-871a-40bc-86e7-022941340d89-maas-api-openshift-ca\") pod \"0c603a9a-871a-40bc-86e7-022941340d89\" (UID: \"0c603a9a-871a-40bc-86e7-022941340d89\") " Oct 31 00:45:19.496573 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:19.496536 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c603a9a-871a-40bc-86e7-022941340d89-kube-api-access-d6sl4\" (UniqueName: \"kubernetes.io/projected/0c603a9a-871a-40bc-86e7-022941340d89-kube-api-access-d6sl4\") pod \"0c603a9a-871a-40bc-86e7-022941340d89\" (UID: \"0c603a9a-871a-40bc-86e7-022941340d89\") " Oct 31 00:45:19.496815 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:19.496629 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c603a9a-871a-40bc-86e7-022941340d89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c603a9a-871a-40bc-86e7-022941340d89-maas-api-xks-ca\") pod \"0c603a9a-871a-40bc-86e7-022941340d89\" (UID: \"0c603a9a-871a-40bc-86e7-022941340d89\") " Oct 31 00:45:19.496894 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:19.496871 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c603a9a-871a-40bc-86e7-022941340d89-maas-api-openshift-ca" pod "0c603a9a-871a-40bc-86e7-022941340d89" (UID: "0c603a9a-871a-40bc-86e7-022941340d89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:19.498678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:19.498648 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c603a9a-871a-40bc-86e7-022941340d89-maas-api-xks-ca" pod "0c603a9a-871a-40bc-86e7-022941340d89" (UID: "0c603a9a-871a-40bc-86e7-022941340d89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:19.498785 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:19.498733 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c603a9a-871a-40bc-86e7-022941340d89-kube-api-access-d6sl4" pod "0c603a9a-871a-40bc-86e7-022941340d89" (UID: "0c603a9a-871a-40bc-86e7-022941340d89"). InnerVolumeSpecName "kube-api-access-d6sl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:19.597563 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:19.597529 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c603a9a-871a-40bc-86e7-022941340d89-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:19.597563 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:19.597557 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6sl4\" (UniqueName: \"kubernetes.io/projected/0c603a9a-871a-40bc-86e7-022941340d89-kube-api-access-d6sl4\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:19.597563 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:19.597566 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c603a9a-871a-40bc-86e7-022941340d89-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:19.646968 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:19.646936 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:20.095764 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:20.095732 4519 scope.go:122] "RemoveContainer" containerID="c2ce7b2fda52e4e1b293d8e06068e22aab5473f33a90d286ffc43015cff1c460" Oct 31 00:45:20.095973 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:20.095856 4519 scope.go:122] "RemoveContainer" containerID="105545f29b726128c7d2c8713ca5f286f509a849eca8f4adcaf81668d2eb7ceb" Oct 31 00:45:20.241450 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:20.241418 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp" event={"ID":"0c603a9a-871a-40bc-86e7-022941340d89","Type":"ContainerDied","Data":"93542ff898414d6795eaf5f2bf4452da8a1b08a34c4add3f58e6548f6559d301"} Oct 31 00:45:20.241858 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:20.241454 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp" Oct 31 00:45:20.241858 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:20.241464 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93542ff898414d6795eaf5f2bf4452da8a1b08a34c4add3f58e6548f6559d301" Oct 31 00:45:20.545578 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:20.545545 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:21.227337 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:21.227305 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:21.249242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:21.249214 4519 generic.go:356] "Generic (PLEG): container finished" podID="84dc7f22-aca3-4a19-9805-1b5c8fa92863" containerID="848ad161ae77e21370c4b409ed29f0c902f073acc73c85e38aeb456dda6d3b60" exitCode=60 Oct 31 00:45:21.249706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:21.249287 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" event={"ID":"84dc7f22-aca3-4a19-9805-1b5c8fa92863","Type":"ContainerDied","Data":"848ad161ae77e21370c4b409ed29f0c902f073acc73c85e38aeb456dda6d3b60"} Oct 31 00:45:21.249706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:21.249343 4519 scope.go:122] "RemoveContainer" containerID="105545f29b726128c7d2c8713ca5f286f509a849eca8f4adcaf81668d2eb7ceb" Oct 31 00:45:21.249816 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:21.249741 4519 scope.go:122] "RemoveContainer" containerID="848ad161ae77e21370c4b409ed29f0c902f073acc73c85e38aeb456dda6d3b60" Oct 31 00:45:21.250045 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:45:21.250012 4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d_odh-ai-gateway-infra(84dc7f22-aca3-4a19-9805-1b5c8fa92863)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" podUID="84dc7f22-aca3-4a19-9805-1b5c8fa92863" Oct 31 00:45:21.251422 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:21.251404 4519 generic.go:356] "Generic (PLEG): container finished" podID="f8a53a21-ed7f-4c46-aad3-4b9c68733d01" containerID="4352b9b69b644b573de4c66b6f5a25618b6089dc132b6d101420a5876dcc6594" exitCode=60 Oct 31 00:45:21.251516 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:21.251436 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" event={"ID":"f8a53a21-ed7f-4c46-aad3-4b9c68733d01","Type":"ContainerDied","Data":"4352b9b69b644b573de4c66b6f5a25618b6089dc132b6d101420a5876dcc6594"} Oct 31 00:45:21.251743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:21.251728 4519 scope.go:122] "RemoveContainer" containerID="4352b9b69b644b573de4c66b6f5a25618b6089dc132b6d101420a5876dcc6594" Oct 31 00:45:21.251959 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:45:21.251943 4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w_odh-ai-gateway-infra(f8a53a21-ed7f-4c46-aad3-4b9c68733d01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" podUID="f8a53a21-ed7f-4c46-aad3-4b9c68733d01" Oct 31 00:45:21.261196 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:21.261175 4519 scope.go:122] "RemoveContainer" containerID="c2ce7b2fda52e4e1b293d8e06068e22aab5473f33a90d286ffc43015cff1c460" Oct 31 00:45:21.845440 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:21.845408 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:22.095573 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.095482 4519 scope.go:122] "RemoveContainer" containerID="aba6c40f302a8e4262b0c169bdce8f5ae909d411a2474ab5fbeba0da9750a593" Oct 31 00:45:22.275563 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.275533 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d"] Oct 31 00:45:22.291139 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.291111 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w"] Oct 31 00:45:22.427092 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.427065 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" Oct 31 00:45:22.468956 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.468934 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" Oct 31 00:45:22.494054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.494005 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:22.534335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.534299 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84dc7f22-aca3-4a19-9805-1b5c8fa92863-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84dc7f22-aca3-4a19-9805-1b5c8fa92863-maas-api-service-ca\") pod \"84dc7f22-aca3-4a19-9805-1b5c8fa92863\" (UID: \"84dc7f22-aca3-4a19-9805-1b5c8fa92863\") " Oct 31 00:45:22.534551 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.534350 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84dc7f22-aca3-4a19-9805-1b5c8fa92863-kube-api-access-bdqsj\" (UniqueName: \"kubernetes.io/projected/84dc7f22-aca3-4a19-9805-1b5c8fa92863-kube-api-access-bdqsj\") pod \"84dc7f22-aca3-4a19-9805-1b5c8fa92863\" (UID: \"84dc7f22-aca3-4a19-9805-1b5c8fa92863\") " Oct 31 00:45:22.535008 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.534979 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84dc7f22-aca3-4a19-9805-1b5c8fa92863-maas-api-service-ca" pod "84dc7f22-aca3-4a19-9805-1b5c8fa92863" (UID: "84dc7f22-aca3-4a19-9805-1b5c8fa92863"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:22.539823 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.539746 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a53a21-ed7f-4c46-aad3-4b9c68733d01-kube-api-access-krd69\" (UniqueName: \"kubernetes.io/projected/f8a53a21-ed7f-4c46-aad3-4b9c68733d01-kube-api-access-krd69\") pod \"f8a53a21-ed7f-4c46-aad3-4b9c68733d01\" (UID: \"f8a53a21-ed7f-4c46-aad3-4b9c68733d01\") " Oct 31 00:45:22.547964 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.547620 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8a53a21-ed7f-4c46-aad3-4b9c68733d01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a53a21-ed7f-4c46-aad3-4b9c68733d01-maas-api-service-ca\") pod \"f8a53a21-ed7f-4c46-aad3-4b9c68733d01\" (UID: \"f8a53a21-ed7f-4c46-aad3-4b9c68733d01\") " Oct 31 00:45:22.548120 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.547932 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a53a21-ed7f-4c46-aad3-4b9c68733d01-maas-api-service-ca" pod "f8a53a21-ed7f-4c46-aad3-4b9c68733d01" (UID: "f8a53a21-ed7f-4c46-aad3-4b9c68733d01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:22.548290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.548274 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a53a21-ed7f-4c46-aad3-4b9c68733d01-maas-api-service-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:22.548353 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.548292 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84dc7f22-aca3-4a19-9805-1b5c8fa92863-maas-api-service-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:22.557253 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.557044 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a53a21-ed7f-4c46-aad3-4b9c68733d01-kube-api-access-krd69" pod "f8a53a21-ed7f-4c46-aad3-4b9c68733d01" (UID: "f8a53a21-ed7f-4c46-aad3-4b9c68733d01"). InnerVolumeSpecName "kube-api-access-krd69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:22.557598 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.557559 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84dc7f22-aca3-4a19-9805-1b5c8fa92863-kube-api-access-bdqsj" pod "84dc7f22-aca3-4a19-9805-1b5c8fa92863" (UID: "84dc7f22-aca3-4a19-9805-1b5c8fa92863"). InnerVolumeSpecName "kube-api-access-bdqsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:22.648908 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.648832 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdqsj\" (UniqueName: \"kubernetes.io/projected/84dc7f22-aca3-4a19-9805-1b5c8fa92863-kube-api-access-bdqsj\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:22.648908 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:22.648861 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krd69\" (UniqueName: \"kubernetes.io/projected/f8a53a21-ed7f-4c46-aad3-4b9c68733d01-kube-api-access-krd69\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:23.126827 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:23.126793 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:23.266340 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:23.266312 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" Oct 31 00:45:23.266522 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:23.266311 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w" event={"ID":"f8a53a21-ed7f-4c46-aad3-4b9c68733d01","Type":"ContainerDied","Data":"0d5990949eadbfaad209801a91d225b5385e256883e6a5f317ceed6ac211055f"} Oct 31 00:45:23.266522 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:23.266437 4519 scope.go:122] "RemoveContainer" containerID="4352b9b69b644b573de4c66b6f5a25618b6089dc132b6d101420a5876dcc6594" Oct 31 00:45:23.267862 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:23.267833 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" Oct 31 00:45:23.267971 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:23.267859 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d" event={"ID":"84dc7f22-aca3-4a19-9805-1b5c8fa92863","Type":"ContainerDied","Data":"039683fa9dae906646b9433550e15545f667c351b76daee9200dd0c20ac46819"} Oct 31 00:45:23.269894 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:23.269871 4519 generic.go:356] "Generic (PLEG): container finished" podID="a24a1307-b598-4f8f-860c-d59bb860c3c7" containerID="6698290f3d28e21a8a81cf9d08e0414b7942716b2efa545e942a4a6bf53aaaac" exitCode=60 Oct 31 00:45:23.270013 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:23.269905 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" event={"ID":"a24a1307-b598-4f8f-860c-d59bb860c3c7","Type":"ContainerDied","Data":"6698290f3d28e21a8a81cf9d08e0414b7942716b2efa545e942a4a6bf53aaaac"} Oct 31 00:45:23.270258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:23.270243 4519 scope.go:122] "RemoveContainer" containerID="6698290f3d28e21a8a81cf9d08e0414b7942716b2efa545e942a4a6bf53aaaac" Oct 31 00:45:23.270497 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:45:23.270478 4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl_odh-ai-gateway-infra(a24a1307-b598-4f8f-860c-d59bb860c3c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" podUID="a24a1307-b598-4f8f-860c-d59bb860c3c7" Oct 31 00:45:23.283719 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:23.283700 4519 scope.go:122] "RemoveContainer" containerID="848ad161ae77e21370c4b409ed29f0c902f073acc73c85e38aeb456dda6d3b60" Oct 31 00:45:23.291963 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:23.291946 4519 scope.go:122] "RemoveContainer" containerID="aba6c40f302a8e4262b0c169bdce8f5ae909d411a2474ab5fbeba0da9750a593" Oct 31 00:45:23.315871 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:23.315846 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w"] Oct 31 00:45:23.318644 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:23.318611 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-47f370-29890125-rcn8w"] Oct 31 00:45:23.331713 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:23.331691 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d"] Oct 31 00:45:23.337523 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:23.337503 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7f1496-29890125-wzj8d"] Oct 31 00:45:23.838053 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:23.838020 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:24.102668 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:24.102578 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84dc7f22-aca3-4a19-9805-1b5c8fa92863" path="/var/lib/kubelet/pods/84dc7f22-aca3-4a19-9805-1b5c8fa92863/volumes" Oct 31 00:45:24.103016 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:24.103001 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a53a21-ed7f-4c46-aad3-4b9c68733d01" path="/var/lib/kubelet/pods/f8a53a21-ed7f-4c46-aad3-4b9c68733d01/volumes" Oct 31 00:45:24.310174 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:24.310137 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl"] Oct 31 00:45:24.450360 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:24.450335 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" Oct 31 00:45:24.542712 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:24.542684 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:24.568197 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:24.568168 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a24a1307-b598-4f8f-860c-d59bb860c3c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24a1307-b598-4f8f-860c-d59bb860c3c7-maas-api-service-ca\") pod \"a24a1307-b598-4f8f-860c-d59bb860c3c7\" (UID: \"a24a1307-b598-4f8f-860c-d59bb860c3c7\") " Oct 31 00:45:24.568353 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:24.568211 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a24a1307-b598-4f8f-860c-d59bb860c3c7-kube-api-access-j6vj4\" (UniqueName: \"kubernetes.io/projected/a24a1307-b598-4f8f-860c-d59bb860c3c7-kube-api-access-j6vj4\") pod \"a24a1307-b598-4f8f-860c-d59bb860c3c7\" (UID: \"a24a1307-b598-4f8f-860c-d59bb860c3c7\") " Oct 31 00:45:24.568580 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:24.568559 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a24a1307-b598-4f8f-860c-d59bb860c3c7-maas-api-service-ca" pod "a24a1307-b598-4f8f-860c-d59bb860c3c7" (UID: "a24a1307-b598-4f8f-860c-d59bb860c3c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:24.570236 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:24.570218 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24a1307-b598-4f8f-860c-d59bb860c3c7-kube-api-access-j6vj4" pod "a24a1307-b598-4f8f-860c-d59bb860c3c7" (UID: "a24a1307-b598-4f8f-860c-d59bb860c3c7"). InnerVolumeSpecName "kube-api-access-j6vj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:24.669699 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:24.669609 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24a1307-b598-4f8f-860c-d59bb860c3c7-maas-api-service-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:24.669699 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:24.669652 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6vj4\" (UniqueName: \"kubernetes.io/projected/a24a1307-b598-4f8f-860c-d59bb860c3c7-kube-api-access-j6vj4\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:25.291395 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:25.291362 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" event={"ID":"a24a1307-b598-4f8f-860c-d59bb860c3c7","Type":"ContainerDied","Data":"c6ee5288d1245a43034e5331a8347193a569ae96722532541eec6974d964ee21"} Oct 31 00:45:25.291587 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:25.291412 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl" Oct 31 00:45:25.291587 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:25.291426 4519 scope.go:122] "RemoveContainer" containerID="6698290f3d28e21a8a81cf9d08e0414b7942716b2efa545e942a4a6bf53aaaac" Oct 31 00:45:25.330853 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:25.330819 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl"] Oct 31 00:45:25.340123 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:25.340092 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1d4a7e-29890125-c87cl"] Oct 31 00:45:25.502904 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:25.502867 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:26.101763 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:26.101732 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a24a1307-b598-4f8f-860c-d59bb860c3c7" path="/var/lib/kubelet/pods/a24a1307-b598-4f8f-860c-d59bb860c3c7/volumes" Oct 31 00:45:26.362384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:26.362281 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:27.357152 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:27.357113 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:28.151899 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:28.151865 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:28.846040 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:28.846000 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:29.810351 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.810307 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:29.885678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.885619 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l"] Oct 31 00:45:29.887088 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.887056 4519 cpu_manager.go:414] "containerMap: removing container" podUID="a24a1307-b598-4f8f-860c-d59bb860c3c7" containerName="cleanup" Oct 31 00:45:29.887805 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.887776 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a24a1307-b598-4f8f-860c-d59bb860c3c7" containerName="cleanup" Oct 31 00:45:29.888014 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.887998 4519 cpu_manager.go:414] "containerMap: removing container" podUID="84dc7f22-aca3-4a19-9805-1b5c8fa92863" containerName="cleanup" Oct 31 00:45:29.903075 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.902980 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84dc7f22-aca3-4a19-9805-1b5c8fa92863" containerName="cleanup" Oct 31 00:45:29.904165 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.904143 4519 cpu_manager.go:414] "containerMap: removing container" podUID="f8a53a21-ed7f-4c46-aad3-4b9c68733d01" containerName="cleanup" Oct 31 00:45:29.904315 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.904302 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a53a21-ed7f-4c46-aad3-4b9c68733d01" containerName="cleanup" Oct 31 00:45:29.904410 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.904399 4519 cpu_manager.go:414] "containerMap: removing container" podUID="f8a53a21-ed7f-4c46-aad3-4b9c68733d01" containerName="cleanup" Oct 31 00:45:29.904505 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.904494 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a53a21-ed7f-4c46-aad3-4b9c68733d01" containerName="cleanup" Oct 31 00:45:29.904671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.904656 4519 cpu_manager.go:414] "containerMap: removing container" podUID="6dfb7609-1a91-4376-90d1-b4baf6047398" containerName="revoke-keys" Oct 31 00:45:29.904796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.904782 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dfb7609-1a91-4376-90d1-b4baf6047398" containerName="revoke-keys" Oct 31 00:45:29.910257 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.910223 4519 cpu_manager.go:414] "containerMap: removing container" podUID="a24a1307-b598-4f8f-860c-d59bb860c3c7" containerName="cleanup" Oct 31 00:45:29.910257 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.910255 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a24a1307-b598-4f8f-860c-d59bb860c3c7" containerName="cleanup" Oct 31 00:45:29.910482 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.910274 4519 cpu_manager.go:414] "containerMap: removing container" podUID="0c603a9a-871a-40bc-86e7-022941340d89" containerName="revoke-keys" Oct 31 00:45:29.910482 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.910283 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c603a9a-871a-40bc-86e7-022941340d89" containerName="revoke-keys" Oct 31 00:45:29.910482 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.910298 4519 cpu_manager.go:414] "containerMap: removing container" podUID="84dc7f22-aca3-4a19-9805-1b5c8fa92863" containerName="cleanup" Oct 31 00:45:29.910482 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.910305 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84dc7f22-aca3-4a19-9805-1b5c8fa92863" containerName="cleanup" Oct 31 00:45:29.910482 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.910452 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a24a1307-b598-4f8f-860c-d59bb860c3c7" containerName="cleanup" Oct 31 00:45:29.910482 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.910465 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="84dc7f22-aca3-4a19-9805-1b5c8fa92863" containerName="cleanup" Oct 31 00:45:29.910482 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.910477 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a53a21-ed7f-4c46-aad3-4b9c68733d01" containerName="cleanup" Oct 31 00:45:29.910482 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.910487 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a53a21-ed7f-4c46-aad3-4b9c68733d01" containerName="cleanup" Oct 31 00:45:29.910929 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.910499 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a53a21-ed7f-4c46-aad3-4b9c68733d01" containerName="cleanup" Oct 31 00:45:29.910929 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.910508 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dfb7609-1a91-4376-90d1-b4baf6047398" containerName="revoke-keys" Oct 31 00:45:29.910929 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.910518 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a24a1307-b598-4f8f-860c-d59bb860c3c7" containerName="cleanup" Oct 31 00:45:29.910929 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.910527 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c603a9a-871a-40bc-86e7-022941340d89" containerName="revoke-keys" Oct 31 00:45:29.910929 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.910535 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="84dc7f22-aca3-4a19-9805-1b5c8fa92863" containerName="cleanup" Oct 31 00:45:29.917567 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.916582 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l"] Oct 31 00:45:29.917567 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.916753 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" Oct 31 00:45:29.921502 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.921475 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:45:29.922840 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.922811 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:45:29.932735 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.932697 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65862e2f-a808-4b28-a903-a1defecc2a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32426f48-xxf4l\" (UID: \"65862e2f-a808-4b28-a903-a1defecc2a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" Oct 31 00:45:29.932877 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.932750 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj9k9\" (UniqueName: \"kubernetes.io/projected/65862e2f-a808-4b28-a903-a1defecc2a75-kube-api-access-rj9k9\") pod \"maas-api-revoke-sub-32426f48-xxf4l\" (UID: \"65862e2f-a808-4b28-a903-a1defecc2a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" Oct 31 00:45:29.932877 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:29.932863 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65862e2f-a808-4b28-a903-a1defecc2a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32426f48-xxf4l\" (UID: \"65862e2f-a808-4b28-a903-a1defecc2a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" Oct 31 00:45:30.033823 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:30.033789 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj9k9\" (UniqueName: \"kubernetes.io/projected/65862e2f-a808-4b28-a903-a1defecc2a75-kube-api-access-rj9k9\") pod \"maas-api-revoke-sub-32426f48-xxf4l\" (UID: \"65862e2f-a808-4b28-a903-a1defecc2a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" Oct 31 00:45:30.033823 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:30.033834 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65862e2f-a808-4b28-a903-a1defecc2a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32426f48-xxf4l\" (UID: \"65862e2f-a808-4b28-a903-a1defecc2a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" Oct 31 00:45:30.034047 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:30.033938 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65862e2f-a808-4b28-a903-a1defecc2a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32426f48-xxf4l\" (UID: \"65862e2f-a808-4b28-a903-a1defecc2a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" Oct 31 00:45:30.034583 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:30.034556 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65862e2f-a808-4b28-a903-a1defecc2a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32426f48-xxf4l\" (UID: \"65862e2f-a808-4b28-a903-a1defecc2a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" Oct 31 00:45:30.036499 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:30.036474 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65862e2f-a808-4b28-a903-a1defecc2a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32426f48-xxf4l\" (UID: \"65862e2f-a808-4b28-a903-a1defecc2a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" Oct 31 00:45:30.041977 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:30.041954 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj9k9\" (UniqueName: \"kubernetes.io/projected/65862e2f-a808-4b28-a903-a1defecc2a75-kube-api-access-rj9k9\") pod \"maas-api-revoke-sub-32426f48-xxf4l\" (UID: \"65862e2f-a808-4b28-a903-a1defecc2a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" Oct 31 00:45:30.233110 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:30.233071 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" Oct 31 00:45:30.538424 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:30.538385 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l"] Oct 31 00:45:30.539457 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:45:30.539429 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65862e2f_a808_4b28_a903_a1defecc2a75.slice/crio-dea6b12806fb7aee4e5ee2eb8725ad556ef0ce30da7cacfc0b424c6c6c6aa334 WatchSource:0}: Error finding container dea6b12806fb7aee4e5ee2eb8725ad556ef0ce30da7cacfc0b424c6c6c6aa334: Status 404 returned error can't find the container with id dea6b12806fb7aee4e5ee2eb8725ad556ef0ce30da7cacfc0b424c6c6c6aa334 Oct 31 00:45:30.752329 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:30.752291 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:31.150522 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.150442 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd"] Oct 31 00:45:31.151075 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.151055 4519 cpu_manager.go:414] "containerMap: removing container" podUID="a24a1307-b598-4f8f-860c-d59bb860c3c7" containerName="cleanup" Oct 31 00:45:31.151147 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.151077 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a24a1307-b598-4f8f-860c-d59bb860c3c7" containerName="cleanup" Oct 31 00:45:31.151147 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.151092 4519 cpu_manager.go:414] "containerMap: removing container" podUID="84dc7f22-aca3-4a19-9805-1b5c8fa92863" containerName="cleanup" Oct 31 00:45:31.151147 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.151099 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84dc7f22-aca3-4a19-9805-1b5c8fa92863" containerName="cleanup" Oct 31 00:45:31.151147 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.151123 4519 cpu_manager.go:414] "containerMap: removing container" podUID="f8a53a21-ed7f-4c46-aad3-4b9c68733d01" containerName="cleanup" Oct 31 00:45:31.151147 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.151130 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a53a21-ed7f-4c46-aad3-4b9c68733d01" containerName="cleanup" Oct 31 00:45:31.151390 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.151254 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a24a1307-b598-4f8f-860c-d59bb860c3c7" containerName="cleanup" Oct 31 00:45:31.151390 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.151265 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="84dc7f22-aca3-4a19-9805-1b5c8fa92863" containerName="cleanup" Oct 31 00:45:31.154361 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.154343 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" Oct 31 00:45:31.161338 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.161312 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd"] Oct 31 00:45:31.246288 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.246257 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c94lt\" (UniqueName: \"kubernetes.io/projected/edd43374-f0a4-4e87-ab46-cac2f75281cd-kube-api-access-c94lt\") pod \"maas-api-revoke-sub-3817b7d5-zdxkd\" (UID: \"edd43374-f0a4-4e87-ab46-cac2f75281cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" Oct 31 00:45:31.246470 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.246334 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edd43374-f0a4-4e87-ab46-cac2f75281cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3817b7d5-zdxkd\" (UID: \"edd43374-f0a4-4e87-ab46-cac2f75281cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" Oct 31 00:45:31.246470 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.246380 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edd43374-f0a4-4e87-ab46-cac2f75281cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3817b7d5-zdxkd\" (UID: \"edd43374-f0a4-4e87-ab46-cac2f75281cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" Oct 31 00:45:31.325971 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.325927 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" event={"ID":"65862e2f-a808-4b28-a903-a1defecc2a75","Type":"ContainerStarted","Data":"b813adb67d791046ec39734c8e109578e131b3d900a29482f238258644d7d56e"} Oct 31 00:45:31.326150 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.325990 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" event={"ID":"65862e2f-a808-4b28-a903-a1defecc2a75","Type":"ContainerStarted","Data":"dea6b12806fb7aee4e5ee2eb8725ad556ef0ce30da7cacfc0b424c6c6c6aa334"} Oct 31 00:45:31.344249 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.344198 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" podStartSLOduration=2.344183439 podStartE2EDuration="2.344183439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:31.343150876 +0000 UTC m=+727.372331270" watchObservedRunningTime="2026-10-31 00:45:31.344183439 +0000 UTC m=+727.373363828" Oct 31 00:45:31.347903 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.347867 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edd43374-f0a4-4e87-ab46-cac2f75281cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3817b7d5-zdxkd\" (UID: \"edd43374-f0a4-4e87-ab46-cac2f75281cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" Oct 31 00:45:31.348040 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.347990 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c94lt\" (UniqueName: \"kubernetes.io/projected/edd43374-f0a4-4e87-ab46-cac2f75281cd-kube-api-access-c94lt\") pod \"maas-api-revoke-sub-3817b7d5-zdxkd\" (UID: \"edd43374-f0a4-4e87-ab46-cac2f75281cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" Oct 31 00:45:31.348093 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.348057 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edd43374-f0a4-4e87-ab46-cac2f75281cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3817b7d5-zdxkd\" (UID: \"edd43374-f0a4-4e87-ab46-cac2f75281cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" Oct 31 00:45:31.348518 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.348499 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edd43374-f0a4-4e87-ab46-cac2f75281cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3817b7d5-zdxkd\" (UID: \"edd43374-f0a4-4e87-ab46-cac2f75281cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" Oct 31 00:45:31.350421 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.350401 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edd43374-f0a4-4e87-ab46-cac2f75281cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3817b7d5-zdxkd\" (UID: \"edd43374-f0a4-4e87-ab46-cac2f75281cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" Oct 31 00:45:31.355925 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.355903 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c94lt\" (UniqueName: \"kubernetes.io/projected/edd43374-f0a4-4e87-ab46-cac2f75281cd-kube-api-access-c94lt\") pod \"maas-api-revoke-sub-3817b7d5-zdxkd\" (UID: \"edd43374-f0a4-4e87-ab46-cac2f75281cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" Oct 31 00:45:31.429008 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.428880 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:31.471263 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.471233 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" Oct 31 00:45:31.516698 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.516657 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:31.751996 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:31.751970 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd"] Oct 31 00:45:31.753140 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:45:31.753115 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd43374_f0a4_4e87_ab46_cac2f75281cd.slice/crio-1548c32bb563895c574105bd80dab3bbb299e4f8bb0ccb3b7ad9042ef07e6fdd WatchSource:0}: Error finding container 1548c32bb563895c574105bd80dab3bbb299e4f8bb0ccb3b7ad9042ef07e6fdd: Status 404 returned error can't find the container with id 1548c32bb563895c574105bd80dab3bbb299e4f8bb0ccb3b7ad9042ef07e6fdd Oct 31 00:45:32.332527 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:32.332493 4519 generic.go:356] "Generic (PLEG): container finished" podID="65862e2f-a808-4b28-a903-a1defecc2a75" containerID="b813adb67d791046ec39734c8e109578e131b3d900a29482f238258644d7d56e" exitCode=0 Oct 31 00:45:32.332970 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:32.332585 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" event={"ID":"65862e2f-a808-4b28-a903-a1defecc2a75","Type":"ContainerDied","Data":"b813adb67d791046ec39734c8e109578e131b3d900a29482f238258644d7d56e"} Oct 31 00:45:32.334204 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:32.334184 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" event={"ID":"edd43374-f0a4-4e87-ab46-cac2f75281cd","Type":"ContainerStarted","Data":"11d440a8f9eaa393d80dd02fc33acc8d198d4b765515323b7823f437ef09e30a"} Oct 31 00:45:32.334293 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:32.334220 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" event={"ID":"edd43374-f0a4-4e87-ab46-cac2f75281cd","Type":"ContainerStarted","Data":"1548c32bb563895c574105bd80dab3bbb299e4f8bb0ccb3b7ad9042ef07e6fdd"} Oct 31 00:45:32.420749 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:32.420695 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" podStartSLOduration=1.420678002 podStartE2EDuration="1.420678002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:32.419202974 +0000 UTC m=+728.448383376" watchObservedRunningTime="2026-10-31 00:45:32.420678002 +0000 UTC m=+728.449858394" Oct 31 00:45:32.647831 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:32.647801 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:33.340903 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:33.340872 4519 generic.go:356] "Generic (PLEG): container finished" podID="edd43374-f0a4-4e87-ab46-cac2f75281cd" containerID="11d440a8f9eaa393d80dd02fc33acc8d198d4b765515323b7823f437ef09e30a" exitCode=0 Oct 31 00:45:33.341295 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:33.340956 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" event={"ID":"edd43374-f0a4-4e87-ab46-cac2f75281cd","Type":"ContainerDied","Data":"11d440a8f9eaa393d80dd02fc33acc8d198d4b765515323b7823f437ef09e30a"} Oct 31 00:45:33.369136 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:33.369097 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:33.491096 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:33.491063 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" Oct 31 00:45:33.572178 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:33.572137 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65862e2f-a808-4b28-a903-a1defecc2a75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65862e2f-a808-4b28-a903-a1defecc2a75-maas-api-xks-ca\") pod \"65862e2f-a808-4b28-a903-a1defecc2a75\" (UID: \"65862e2f-a808-4b28-a903-a1defecc2a75\") " Oct 31 00:45:33.572360 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:33.572225 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65862e2f-a808-4b28-a903-a1defecc2a75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65862e2f-a808-4b28-a903-a1defecc2a75-maas-api-openshift-ca\") pod \"65862e2f-a808-4b28-a903-a1defecc2a75\" (UID: \"65862e2f-a808-4b28-a903-a1defecc2a75\") " Oct 31 00:45:33.572360 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:33.572323 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65862e2f-a808-4b28-a903-a1defecc2a75-kube-api-access-rj9k9\" (UniqueName: \"kubernetes.io/projected/65862e2f-a808-4b28-a903-a1defecc2a75-kube-api-access-rj9k9\") pod \"65862e2f-a808-4b28-a903-a1defecc2a75\" (UID: \"65862e2f-a808-4b28-a903-a1defecc2a75\") " Oct 31 00:45:33.572555 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:33.572535 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65862e2f-a808-4b28-a903-a1defecc2a75-maas-api-openshift-ca" pod "65862e2f-a808-4b28-a903-a1defecc2a75" (UID: "65862e2f-a808-4b28-a903-a1defecc2a75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:33.572709 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:33.572673 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65862e2f-a808-4b28-a903-a1defecc2a75-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:33.574328 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:33.574309 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65862e2f-a808-4b28-a903-a1defecc2a75-kube-api-access-rj9k9" pod "65862e2f-a808-4b28-a903-a1defecc2a75" (UID: "65862e2f-a808-4b28-a903-a1defecc2a75"). InnerVolumeSpecName "kube-api-access-rj9k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:33.574432 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:33.574411 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65862e2f-a808-4b28-a903-a1defecc2a75-maas-api-xks-ca" pod "65862e2f-a808-4b28-a903-a1defecc2a75" (UID: "65862e2f-a808-4b28-a903-a1defecc2a75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:33.673776 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:33.673700 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65862e2f-a808-4b28-a903-a1defecc2a75-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:33.673776 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:33.673729 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj9k9\" (UniqueName: \"kubernetes.io/projected/65862e2f-a808-4b28-a903-a1defecc2a75-kube-api-access-rj9k9\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:34.211157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:34.211094 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:34.346747 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:34.346718 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" Oct 31 00:45:34.347162 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:34.346759 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l" event={"ID":"65862e2f-a808-4b28-a903-a1defecc2a75","Type":"ContainerDied","Data":"dea6b12806fb7aee4e5ee2eb8725ad556ef0ce30da7cacfc0b424c6c6c6aa334"} Oct 31 00:45:34.347162 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:34.346809 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dea6b12806fb7aee4e5ee2eb8725ad556ef0ce30da7cacfc0b424c6c6c6aa334" Oct 31 00:45:34.485231 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:34.485203 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" Oct 31 00:45:34.583926 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:34.583894 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edd43374-f0a4-4e87-ab46-cac2f75281cd-kube-api-access-c94lt\" (UniqueName: \"kubernetes.io/projected/edd43374-f0a4-4e87-ab46-cac2f75281cd-kube-api-access-c94lt\") pod \"edd43374-f0a4-4e87-ab46-cac2f75281cd\" (UID: \"edd43374-f0a4-4e87-ab46-cac2f75281cd\") " Oct 31 00:45:34.584127 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:34.583961 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edd43374-f0a4-4e87-ab46-cac2f75281cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edd43374-f0a4-4e87-ab46-cac2f75281cd-maas-api-openshift-ca\") pod \"edd43374-f0a4-4e87-ab46-cac2f75281cd\" (UID: \"edd43374-f0a4-4e87-ab46-cac2f75281cd\") " Oct 31 00:45:34.584127 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:34.584002 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edd43374-f0a4-4e87-ab46-cac2f75281cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edd43374-f0a4-4e87-ab46-cac2f75281cd-maas-api-xks-ca\") pod \"edd43374-f0a4-4e87-ab46-cac2f75281cd\" (UID: \"edd43374-f0a4-4e87-ab46-cac2f75281cd\") " Oct 31 00:45:34.584323 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:34.584301 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd43374-f0a4-4e87-ab46-cac2f75281cd-maas-api-openshift-ca" pod "edd43374-f0a4-4e87-ab46-cac2f75281cd" (UID: "edd43374-f0a4-4e87-ab46-cac2f75281cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:34.586098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:34.586070 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edd43374-f0a4-4e87-ab46-cac2f75281cd-maas-api-xks-ca" pod "edd43374-f0a4-4e87-ab46-cac2f75281cd" (UID: "edd43374-f0a4-4e87-ab46-cac2f75281cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:34.586215 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:34.586194 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd43374-f0a4-4e87-ab46-cac2f75281cd-kube-api-access-c94lt" pod "edd43374-f0a4-4e87-ab46-cac2f75281cd" (UID: "edd43374-f0a4-4e87-ab46-cac2f75281cd"). InnerVolumeSpecName "kube-api-access-c94lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:34.685242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:34.685164 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edd43374-f0a4-4e87-ab46-cac2f75281cd-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:34.685242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:34.685202 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c94lt\" (UniqueName: \"kubernetes.io/projected/edd43374-f0a4-4e87-ab46-cac2f75281cd-kube-api-access-c94lt\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:34.685242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:34.685217 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edd43374-f0a4-4e87-ab46-cac2f75281cd-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:34.942369 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:34.942269 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:35.352518 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:35.352491 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" Oct 31 00:45:35.352518 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:35.352506 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd" event={"ID":"edd43374-f0a4-4e87-ab46-cac2f75281cd","Type":"ContainerDied","Data":"1548c32bb563895c574105bd80dab3bbb299e4f8bb0ccb3b7ad9042ef07e6fdd"} Oct 31 00:45:35.352995 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:35.352549 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1548c32bb563895c574105bd80dab3bbb299e4f8bb0ccb3b7ad9042ef07e6fdd" Oct 31 00:45:35.540083 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:35.540055 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:36.232244 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:36.232209 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:36.939365 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:36.939320 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:37.827852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:37.827814 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:38.896871 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:38.896827 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:40.206807 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:40.206778 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:41.168647 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:41.168589 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:42.037450 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.037420 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:42.480040 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.479960 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7"] Oct 31 00:45:42.480395 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.480380 4519 cpu_manager.go:414] "containerMap: removing container" podUID="edd43374-f0a4-4e87-ab46-cac2f75281cd" containerName="revoke-keys" Oct 31 00:45:42.480435 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.480397 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edd43374-f0a4-4e87-ab46-cac2f75281cd" containerName="revoke-keys" Oct 31 00:45:42.480435 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.480418 4519 cpu_manager.go:414] "containerMap: removing container" podUID="65862e2f-a808-4b28-a903-a1defecc2a75" containerName="revoke-keys" Oct 31 00:45:42.480435 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.480423 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65862e2f-a808-4b28-a903-a1defecc2a75" containerName="revoke-keys" Oct 31 00:45:42.480540 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.480502 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="edd43374-f0a4-4e87-ab46-cac2f75281cd" containerName="revoke-keys" Oct 31 00:45:42.480540 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.480512 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="65862e2f-a808-4b28-a903-a1defecc2a75" containerName="revoke-keys" Oct 31 00:45:42.486583 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.486560 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7" Oct 31 00:45:42.491787 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.491757 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7"] Oct 31 00:45:42.492382 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.492348 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:45:42.492500 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.492390 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:45:42.536361 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.536327 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf"] Oct 31 00:45:42.540807 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.540781 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.553154 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.553125 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf"] Oct 31 00:45:42.586846 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.586814 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e1d908f-hrkm7\" (UID: \"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7" Oct 31 00:45:42.587026 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.586990 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e1d908f-hrkm7\" (UID: \"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7" Oct 31 00:45:42.587084 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.587036 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7qsg\" (UniqueName: \"kubernetes.io/projected/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-kube-api-access-g7qsg\") pod \"maas-api-revoke-sub-1e1d908f-hrkm7\" (UID: \"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7" Oct 31 00:45:42.590544 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.590521 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-44bc67-openshift-default-dockercfg-z6hg4\"" Oct 31 00:45:42.688504 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.688451 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-istio-data\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.688504 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.688498 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4320a978-89eb-4601-a10e-028e2549b7dd-istiod-ca-cert\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.688833 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.688529 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e1d908f-hrkm7\" (UID: \"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7" Oct 31 00:45:42.688833 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.688552 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-istio-envoy\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.688833 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.688799 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-workload-certs\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.688991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.688836 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4320a978-89eb-4601-a10e-028e2549b7dd-istio-token\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.688991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.688855 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-workload-socket\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.688991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.688985 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4320a978-89eb-4601-a10e-028e2549b7dd-istio-podinfo\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.689139 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.689030 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e1d908f-hrkm7\" (UID: \"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7" Oct 31 00:45:42.689139 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.689052 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-credential-socket\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.689139 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.689129 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7qsg\" (UniqueName: \"kubernetes.io/projected/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-kube-api-access-g7qsg\") pod \"maas-api-revoke-sub-1e1d908f-hrkm7\" (UID: \"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7" Oct 31 00:45:42.689303 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.689191 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-926r9\" (UniqueName: \"kubernetes.io/projected/4320a978-89eb-4601-a10e-028e2549b7dd-kube-api-access-926r9\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.689733 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.689717 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e1d908f-hrkm7\" (UID: \"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7" Oct 31 00:45:42.691058 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.691035 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e1d908f-hrkm7\" (UID: \"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7" Oct 31 00:45:42.698419 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.698390 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7qsg\" (UniqueName: \"kubernetes.io/projected/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-kube-api-access-g7qsg\") pod \"maas-api-revoke-sub-1e1d908f-hrkm7\" (UID: \"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7" Oct 31 00:45:42.772900 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.772818 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:42.790762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.790733 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-workload-certs\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.790960 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.790776 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4320a978-89eb-4601-a10e-028e2549b7dd-istio-token\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.790960 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.790797 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-workload-socket\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.790960 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.790835 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4320a978-89eb-4601-a10e-028e2549b7dd-istio-podinfo\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.790960 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.790863 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-credential-socket\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.790960 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.790896 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-926r9\" (UniqueName: \"kubernetes.io/projected/4320a978-89eb-4601-a10e-028e2549b7dd-kube-api-access-926r9\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.791275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.791042 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-istio-data\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.791275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.791080 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4320a978-89eb-4601-a10e-028e2549b7dd-istiod-ca-cert\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.791275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.791116 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-istio-envoy\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.791424 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.791266 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-workload-certs\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.791424 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.791355 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-workload-socket\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.791493 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.791416 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-credential-socket\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.791493 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.791482 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-istio-data\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.791757 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.791736 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4320a978-89eb-4601-a10e-028e2549b7dd-istiod-ca-cert\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.793279 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.793250 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-istio-envoy\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.793332 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.793276 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4320a978-89eb-4601-a10e-028e2549b7dd-istio-podinfo\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.804477 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.804451 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7" Oct 31 00:45:42.810985 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.810961 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-926r9\" (UniqueName: \"kubernetes.io/projected/4320a978-89eb-4601-a10e-028e2549b7dd-kube-api-access-926r9\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.811219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.811202 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4320a978-89eb-4601-a10e-028e2549b7dd-istio-token\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:42.856307 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:42.855591 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:43.019623 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.019587 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv"] Oct 31 00:45:43.023831 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.023777 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.031819 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.031792 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f99ac0-openshift-default-dockercfg-5jrvn\"" Oct 31 00:45:43.034546 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.034512 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv"] Oct 31 00:45:43.094900 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.094865 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-workload-certs\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.094900 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.094903 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-credential-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.095427 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.094948 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-podinfo\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.095427 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.095090 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-envoy\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.095427 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.095191 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-workload-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.095427 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.095232 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvbk4\" (UniqueName: \"kubernetes.io/projected/cdbf7b30-65d4-4462-af10-05cdd5015ce3-kube-api-access-dvbk4\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.095427 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.095315 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-token\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.095427 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.095356 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-data\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.095427 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.095409 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istiod-ca-cert\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.125565 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.125537 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7"] Oct 31 00:45:43.126752 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:45:43.126728 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b7dd2d_ce74_4fb7_a39e_a0de7f78b978.slice/crio-4b3ca0be4c7acba6cd5b3f0a30ef9053727781350340530f306c85cbd06fff32 WatchSource:0}: Error finding container 4b3ca0be4c7acba6cd5b3f0a30ef9053727781350340530f306c85cbd06fff32: Status 404 returned error can't find the container with id 4b3ca0be4c7acba6cd5b3f0a30ef9053727781350340530f306c85cbd06fff32 Oct 31 00:45:43.159747 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.159720 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf"] Oct 31 00:45:43.163071 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:45:43.163045 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4320a978_89eb_4601_a10e_028e2549b7dd.slice/crio-ff31bfc0b74f7353275354ca9a4a850a6f028735f4393a48495f168e12a480b0 WatchSource:0}: Error finding container ff31bfc0b74f7353275354ca9a4a850a6f028735f4393a48495f168e12a480b0: Status 404 returned error can't find the container with id ff31bfc0b74f7353275354ca9a4a850a6f028735f4393a48495f168e12a480b0 Oct 31 00:45:43.165371 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.165344 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:43.165444 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.165402 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:43.165444 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.165432 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:43.197051 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.197011 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-podinfo\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.197198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.197071 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-envoy\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.197198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.197148 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-workload-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.197198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.197182 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvbk4\" (UniqueName: \"kubernetes.io/projected/cdbf7b30-65d4-4462-af10-05cdd5015ce3-kube-api-access-dvbk4\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.197373 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.197251 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-token\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.197373 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.197296 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-data\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.197373 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.197327 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istiod-ca-cert\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.197373 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.197363 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-workload-certs\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.197582 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.197392 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-credential-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.197693 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.197668 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-workload-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.197806 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.197785 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-data\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.197872 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.197855 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-credential-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.197930 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.197884 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-workload-certs\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.198058 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.198039 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istiod-ca-cert\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.199553 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.199529 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-podinfo\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.199665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.199624 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-envoy\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.206580 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.206558 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvbk4\" (UniqueName: \"kubernetes.io/projected/cdbf7b30-65d4-4462-af10-05cdd5015ce3-kube-api-access-dvbk4\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.206727 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.206705 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-token\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.245335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.245299 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv"] Oct 31 00:45:43.249955 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.249926 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.265780 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.264834 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv"] Oct 31 00:45:43.340423 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.340341 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:43.401487 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.401172 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d1c38d-9d1d-4674-82cc-38782f2a4f39-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.401487 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.401221 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d1c38d-9d1d-4674-82cc-38782f2a4f39-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.401487 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.401406 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d1c38d-9d1d-4674-82cc-38782f2a4f39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.401487 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.401448 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d1c38d-9d1d-4674-82cc-38782f2a4f39-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.401690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.401523 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d1c38d-9d1d-4674-82cc-38782f2a4f39-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.401690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.401552 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d1c38d-9d1d-4674-82cc-38782f2a4f39-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.401690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.401629 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm8lv\" (UniqueName: \"kubernetes.io/projected/23d1c38d-9d1d-4674-82cc-38782f2a4f39-kube-api-access-zm8lv\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.401690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.401682 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d1c38d-9d1d-4674-82cc-38782f2a4f39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.401892 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.401715 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d1c38d-9d1d-4674-82cc-38782f2a4f39-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.407859 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.407821 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7" event={"ID":"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978","Type":"ContainerStarted","Data":"4b3ca0be4c7acba6cd5b3f0a30ef9053727781350340530f306c85cbd06fff32"} Oct 31 00:45:43.411005 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.410976 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" event={"ID":"4320a978-89eb-4601-a10e-028e2549b7dd","Type":"ContainerStarted","Data":"982cddf7e227e6de5be5793dfd554fe5faf543e77f92aba25cfc581cf4c3957b"} Oct 31 00:45:43.411117 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.411031 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" event={"ID":"4320a978-89eb-4601-a10e-028e2549b7dd","Type":"ContainerStarted","Data":"ff31bfc0b74f7353275354ca9a4a850a6f028735f4393a48495f168e12a480b0"} Oct 31 00:45:43.434846 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.434793 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" podStartSLOduration=1.434776587 podStartE2EDuration="1.434776587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:43.43451469 +0000 UTC m=+739.463695081" watchObservedRunningTime="2026-10-31 00:45:43.434776587 +0000 UTC m=+739.463956968" Oct 31 00:45:43.503085 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.503046 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d1c38d-9d1d-4674-82cc-38782f2a4f39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.503085 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.503084 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d1c38d-9d1d-4674-82cc-38782f2a4f39-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.503306 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.503149 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d1c38d-9d1d-4674-82cc-38782f2a4f39-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.503306 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.503174 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d1c38d-9d1d-4674-82cc-38782f2a4f39-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.503306 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.503258 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm8lv\" (UniqueName: \"kubernetes.io/projected/23d1c38d-9d1d-4674-82cc-38782f2a4f39-kube-api-access-zm8lv\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.503306 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.503292 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d1c38d-9d1d-4674-82cc-38782f2a4f39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.503520 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.503325 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d1c38d-9d1d-4674-82cc-38782f2a4f39-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.503520 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.503412 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d1c38d-9d1d-4674-82cc-38782f2a4f39-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.503520 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.503444 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d1c38d-9d1d-4674-82cc-38782f2a4f39-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.503520 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.503510 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d1c38d-9d1d-4674-82cc-38782f2a4f39-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.503759 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.503578 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d1c38d-9d1d-4674-82cc-38782f2a4f39-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.503895 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.503872 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d1c38d-9d1d-4674-82cc-38782f2a4f39-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.503950 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.503923 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d1c38d-9d1d-4674-82cc-38782f2a4f39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.504030 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.504009 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d1c38d-9d1d-4674-82cc-38782f2a4f39-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.506485 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.506462 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d1c38d-9d1d-4674-82cc-38782f2a4f39-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.506936 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.506915 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d1c38d-9d1d-4674-82cc-38782f2a4f39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.512038 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.512016 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d1c38d-9d1d-4674-82cc-38782f2a4f39-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.512600 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.512579 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm8lv\" (UniqueName: \"kubernetes.io/projected/23d1c38d-9d1d-4674-82cc-38782f2a4f39-kube-api-access-zm8lv\") pod \"maas-default-gateway-openshift-default-6f54779779-457nv\" (UID: \"23d1c38d-9d1d-4674-82cc-38782f2a4f39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.575152 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.575116 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:43.624878 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.624792 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv"] Oct 31 00:45:43.629782 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:45:43.629736 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbf7b30_65d4_4462_af10_05cdd5015ce3.slice/crio-07dfc0550767c8439c5eae61cb9dd3f627e735b180aa1054ba4b607a49cb3805 WatchSource:0}: Error finding container 07dfc0550767c8439c5eae61cb9dd3f627e735b180aa1054ba4b607a49cb3805: Status 404 returned error can't find the container with id 07dfc0550767c8439c5eae61cb9dd3f627e735b180aa1054ba4b607a49cb3805 Oct 31 00:45:43.635103 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.634030 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:43.635103 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.634125 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:43.635103 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.634170 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:43.655689 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.650617 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:43.857706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.857670 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:43.872798 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.872769 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv"] Oct 31 00:45:43.876198 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:45:43.876143 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d1c38d_9d1d_4674_82cc_38782f2a4f39.slice/crio-35b3d950ae9af5f96c0976a5d61b1884727b20f3fcf256c74fc7bc436537a7ae WatchSource:0}: Error finding container 35b3d950ae9af5f96c0976a5d61b1884727b20f3fcf256c74fc7bc436537a7ae: Status 404 returned error can't find the container with id 35b3d950ae9af5f96c0976a5d61b1884727b20f3fcf256c74fc7bc436537a7ae Oct 31 00:45:43.879048 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.879018 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:43.879131 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.879079 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:43.879131 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:43.879108 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:44.419326 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:44.419226 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" event={"ID":"cdbf7b30-65d4-4462-af10-05cdd5015ce3","Type":"ContainerStarted","Data":"727939f6d98ac00afaa2991e4b057de2077f56d8f187b71aabc26ac40a1d97fb"} Oct 31 00:45:44.419326 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:44.419295 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" event={"ID":"cdbf7b30-65d4-4462-af10-05cdd5015ce3","Type":"ContainerStarted","Data":"07dfc0550767c8439c5eae61cb9dd3f627e735b180aa1054ba4b607a49cb3805"} Oct 31 00:45:44.421060 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:44.421034 4519 generic.go:356] "Generic (PLEG): container finished" podID="41b7dd2d-ce74-4fb7-a39e-a0de7f78b978" containerID="e2f6ceae620ace523f8d69c9f1e3378818809415378c8f76d87a0135f23ab46a" exitCode=0 Oct 31 00:45:44.421169 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:44.421119 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7" event={"ID":"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978","Type":"ContainerDied","Data":"e2f6ceae620ace523f8d69c9f1e3378818809415378c8f76d87a0135f23ab46a"} Oct 31 00:45:44.422603 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:44.422581 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" event={"ID":"23d1c38d-9d1d-4674-82cc-38782f2a4f39","Type":"ContainerStarted","Data":"dea2a53ad8229b110e333b14e5587bdc1335a5464cca401672526c15bf41df7a"} Oct 31 00:45:44.422730 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:44.422656 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" event={"ID":"23d1c38d-9d1d-4674-82cc-38782f2a4f39","Type":"ContainerStarted","Data":"35b3d950ae9af5f96c0976a5d61b1884727b20f3fcf256c74fc7bc436537a7ae"} Oct 31 00:45:44.460660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:44.460390 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" podStartSLOduration=2.4603685889999998 podStartE2EDuration="2.460368589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:44.44908084 +0000 UTC m=+740.478261231" watchObservedRunningTime="2026-10-31 00:45:44.460368589 +0000 UTC m=+740.489548980" Oct 31 00:45:44.489363 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:44.489301 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" podStartSLOduration=1.489281608 podStartE2EDuration="1.489281608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:44.484157911 +0000 UTC m=+740.513338302" watchObservedRunningTime="2026-10-31 00:45:44.489281608 +0000 UTC m=+740.518462000" Oct 31 00:45:44.581659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:44.581473 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:44.597681 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:44.582698 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:44.597681 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:44.592701 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-457nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:45:44.597681 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:44.592750 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" podUID="23d1c38d-9d1d-4674-82cc-38782f2a4f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:45:44.866797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:44.866755 4519 patch_prober.go:29] interesting pod/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:45:44.867091 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:44.867065 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" podUID="4320a978-89eb-4601-a10e-028e2549b7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 00:45:45.160051 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:45.159953 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:45.356659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:45.341601 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:45.357153 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:45.357039 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:45:45.357350 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:45.357323 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" podUID="cdbf7b30-65d4-4462-af10-05cdd5015ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 00:45:45.591659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:45.587321 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-457nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:45:45.591659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:45.587382 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" podUID="23d1c38d-9d1d-4674-82cc-38782f2a4f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:45:45.616664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:45.616013 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:45.870718 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:45.859014 4519 patch_prober.go:29] interesting pod/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:45:45.870718 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:45.859075 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" podUID="4320a978-89eb-4601-a10e-028e2549b7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 00:45:46.344659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.342089 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:45:46.344659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.342142 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" podUID="cdbf7b30-65d4-4462-af10-05cdd5015ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 00:45:46.344659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.343543 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7" Oct 31 00:45:46.409659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.409117 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:46.418724 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.413067 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-kube-api-access-g7qsg\" (UniqueName: \"kubernetes.io/projected/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-kube-api-access-g7qsg\") pod \"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978\" (UID: \"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978\") " Oct 31 00:45:46.418724 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.413159 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-maas-api-openshift-ca\") pod \"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978\" (UID: \"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978\") " Oct 31 00:45:46.418724 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.413234 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-maas-api-xks-ca\") pod \"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978\" (UID: \"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978\") " Oct 31 00:45:46.418724 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.414380 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-maas-api-openshift-ca" pod "41b7dd2d-ce74-4fb7-a39e-a0de7f78b978" (UID: "41b7dd2d-ce74-4fb7-a39e-a0de7f78b978"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:46.418724 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.417158 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-maas-api-xks-ca" pod "41b7dd2d-ce74-4fb7-a39e-a0de7f78b978" (UID: "41b7dd2d-ce74-4fb7-a39e-a0de7f78b978"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:46.466658 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.459844 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-kube-api-access-g7qsg" pod "41b7dd2d-ce74-4fb7-a39e-a0de7f78b978" (UID: "41b7dd2d-ce74-4fb7-a39e-a0de7f78b978"). InnerVolumeSpecName "kube-api-access-g7qsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:46.520662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.520236 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:46.520662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.520268 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:46.520662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.520283 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7qsg\" (UniqueName: \"kubernetes.io/projected/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978-kube-api-access-g7qsg\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:46.590729 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.589173 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-457nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:45:46.590729 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.589226 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" podUID="23d1c38d-9d1d-4674-82cc-38782f2a4f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:45:46.601429 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.599199 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7" event={"ID":"41b7dd2d-ce74-4fb7-a39e-a0de7f78b978","Type":"ContainerDied","Data":"4b3ca0be4c7acba6cd5b3f0a30ef9053727781350340530f306c85cbd06fff32"} Oct 31 00:45:46.601429 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.599258 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b3ca0be4c7acba6cd5b3f0a30ef9053727781350340530f306c85cbd06fff32" Oct 31 00:45:46.601429 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.599316 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7" Oct 31 00:45:46.862714 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.862394 4519 patch_prober.go:29] interesting pod/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:45:46.862714 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:46.862460 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" podUID="4320a978-89eb-4601-a10e-028e2549b7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 00:45:47.341250 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:47.341200 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:45:47.341560 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:47.341529 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" podUID="cdbf7b30-65d4-4462-af10-05cdd5015ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 00:45:47.403660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:47.400818 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:47.584674 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:47.576087 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-457nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:45:47.584674 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:47.576149 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" podUID="23d1c38d-9d1d-4674-82cc-38782f2a4f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:45:47.616288 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:45:47.616194 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:45:47.860681 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:47.858012 4519 patch_prober.go:29] interesting pod/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:45:47.860681 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:47.858077 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" podUID="4320a978-89eb-4601-a10e-028e2549b7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 00:45:48.185671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:48.185616 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:48.349855 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:48.349823 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:45:48.350140 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:48.350096 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" podUID="cdbf7b30-65d4-4462-af10-05cdd5015ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 00:45:48.576377 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:48.576336 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-457nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:45:48.576741 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:48.576701 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" podUID="23d1c38d-9d1d-4674-82cc-38782f2a4f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:45:48.858725 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:48.857980 4519 patch_prober.go:29] interesting pod/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:45:48.858725 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:48.858044 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" podUID="4320a978-89eb-4601-a10e-028e2549b7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 00:45:49.084672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:49.076430 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:49.342660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:49.341033 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:45:49.342660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:49.341094 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" podUID="cdbf7b30-65d4-4462-af10-05cdd5015ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 00:45:49.576074 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:49.576033 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-457nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:45:49.576347 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:49.576319 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" podUID="23d1c38d-9d1d-4674-82cc-38782f2a4f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:45:49.846565 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:49.846525 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:49.859041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:49.859002 4519 patch_prober.go:29] interesting pod/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:45:49.859731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:49.859695 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" podUID="4320a978-89eb-4601-a10e-028e2549b7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 00:45:50.341665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:50.341105 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:45:50.341665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:50.341188 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" podUID="cdbf7b30-65d4-4462-af10-05cdd5015ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 00:45:50.575749 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:50.575708 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-457nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:45:50.575935 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:50.575772 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" podUID="23d1c38d-9d1d-4674-82cc-38782f2a4f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:45:50.732858 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:50.732787 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:50.866731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:50.866697 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:50.867222 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:50.867133 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:50.868036 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:50.868011 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:45:51.582271 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:51.582235 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-457nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:45:51.582568 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:51.582537 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" podUID="23d1c38d-9d1d-4674-82cc-38782f2a4f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:45:52.309547 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:52.309516 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:52.310164 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:52.310008 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:52.311181 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:52.311159 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:45:52.580832 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:52.580746 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:53.324585 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:53.324547 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:53.325255 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:53.325230 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:53.326860 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:53.326838 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" Oct 31 00:45:53.836601 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:53.836570 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:54.668949 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.668908 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd"] Oct 31 00:45:54.669451 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.669364 4519 cpu_manager.go:414] "containerMap: removing container" podUID="41b7dd2d-ce74-4fb7-a39e-a0de7f78b978" containerName="revoke-keys" Oct 31 00:45:54.669451 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.669376 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41b7dd2d-ce74-4fb7-a39e-a0de7f78b978" containerName="revoke-keys" Oct 31 00:45:54.669618 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.669479 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="41b7dd2d-ce74-4fb7-a39e-a0de7f78b978" containerName="revoke-keys" Oct 31 00:45:54.674130 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.674106 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" Oct 31 00:45:54.678410 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.678382 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:45:54.678568 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.678524 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:45:54.680881 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.680820 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd"] Oct 31 00:45:54.764048 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.764009 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1283fbf-f92e-465e-99f6-4582a8022574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdd9d6f7-qdnbd\" (UID: \"a1283fbf-f92e-465e-99f6-4582a8022574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" Oct 31 00:45:54.764048 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.764053 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgpfm\" (UniqueName: \"kubernetes.io/projected/a1283fbf-f92e-465e-99f6-4582a8022574-kube-api-access-xgpfm\") pod \"maas-api-revoke-sub-fdd9d6f7-qdnbd\" (UID: \"a1283fbf-f92e-465e-99f6-4582a8022574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" Oct 31 00:45:54.764302 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.764154 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1283fbf-f92e-465e-99f6-4582a8022574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdd9d6f7-qdnbd\" (UID: \"a1283fbf-f92e-465e-99f6-4582a8022574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" Oct 31 00:45:54.834234 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.834197 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:54.865055 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.865015 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1283fbf-f92e-465e-99f6-4582a8022574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdd9d6f7-qdnbd\" (UID: \"a1283fbf-f92e-465e-99f6-4582a8022574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" Oct 31 00:45:54.865263 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.865162 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgpfm\" (UniqueName: \"kubernetes.io/projected/a1283fbf-f92e-465e-99f6-4582a8022574-kube-api-access-xgpfm\") pod \"maas-api-revoke-sub-fdd9d6f7-qdnbd\" (UID: \"a1283fbf-f92e-465e-99f6-4582a8022574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" Oct 31 00:45:54.865263 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.865221 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1283fbf-f92e-465e-99f6-4582a8022574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdd9d6f7-qdnbd\" (UID: \"a1283fbf-f92e-465e-99f6-4582a8022574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" Oct 31 00:45:54.865957 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.865933 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1283fbf-f92e-465e-99f6-4582a8022574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdd9d6f7-qdnbd\" (UID: \"a1283fbf-f92e-465e-99f6-4582a8022574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" Oct 31 00:45:54.867605 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.867578 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1283fbf-f92e-465e-99f6-4582a8022574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdd9d6f7-qdnbd\" (UID: \"a1283fbf-f92e-465e-99f6-4582a8022574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" Oct 31 00:45:54.874098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.874073 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgpfm\" (UniqueName: \"kubernetes.io/projected/a1283fbf-f92e-465e-99f6-4582a8022574-kube-api-access-xgpfm\") pod \"maas-api-revoke-sub-fdd9d6f7-qdnbd\" (UID: \"a1283fbf-f92e-465e-99f6-4582a8022574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" Oct 31 00:45:54.991680 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:54.991645 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" Oct 31 00:45:55.294424 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:55.294388 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd"] Oct 31 00:45:55.295230 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:45:55.295205 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1283fbf_f92e_465e_99f6_4582a8022574.slice/crio-5cbb0220cdf653da23a43315d7d2f150b5ca8eda5a0e38c70345b7415ab977b1 WatchSource:0}: Error finding container 5cbb0220cdf653da23a43315d7d2f150b5ca8eda5a0e38c70345b7415ab977b1: Status 404 returned error can't find the container with id 5cbb0220cdf653da23a43315d7d2f150b5ca8eda5a0e38c70345b7415ab977b1 Oct 31 00:45:55.554531 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:55.554431 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:55.717474 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:55.717439 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" event={"ID":"a1283fbf-f92e-465e-99f6-4582a8022574","Type":"ContainerStarted","Data":"fa0c0f5a7fc362a695939a6fda65ce4f63a18d3d65090afed4a061e3372d4f70"} Oct 31 00:45:55.717902 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:55.717516 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" event={"ID":"a1283fbf-f92e-465e-99f6-4582a8022574","Type":"ContainerStarted","Data":"5cbb0220cdf653da23a43315d7d2f150b5ca8eda5a0e38c70345b7415ab977b1"} Oct 31 00:45:55.735886 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:55.735838 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" podStartSLOduration=1.735821399 podStartE2EDuration="1.735821399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:55.733985616 +0000 UTC m=+751.763166009" watchObservedRunningTime="2026-10-31 00:45:55.735821399 +0000 UTC m=+751.765001790" Oct 31 00:45:56.163495 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:56.163458 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:56.724126 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:56.724086 4519 generic.go:356] "Generic (PLEG): container finished" podID="a1283fbf-f92e-465e-99f6-4582a8022574" containerID="fa0c0f5a7fc362a695939a6fda65ce4f63a18d3d65090afed4a061e3372d4f70" exitCode=0 Oct 31 00:45:56.724539 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:56.724152 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" event={"ID":"a1283fbf-f92e-465e-99f6-4582a8022574","Type":"ContainerDied","Data":"fa0c0f5a7fc362a695939a6fda65ce4f63a18d3d65090afed4a061e3372d4f70"} Oct 31 00:45:56.804693 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:56.804630 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:56.954690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:56.954651 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:57.442649 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.442603 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:57.686070 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.686038 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6"] Oct 31 00:45:57.689940 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.689917 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.697869 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.697399 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b433e6-openshift-default-dockercfg-72t9t\"" Oct 31 00:45:57.707265 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.707236 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6"] Oct 31 00:45:57.715713 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.715684 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6"] Oct 31 00:45:57.719672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.719625 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.750359 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.750314 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6"] Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.820500 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-credential-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.820552 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-data\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.820676 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-workload-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.820716 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-workload-certs\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.820744 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-credential-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.820763 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-istio-envoy\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.820788 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd4f0b9-e235-40f7-acef-68e1051fc918-istiod-ca-cert\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.820810 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l947c\" (UniqueName: \"kubernetes.io/projected/467bdac4-5015-4338-a6b1-c7591c441847-kube-api-access-l947c\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.820837 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxmkl\" (UniqueName: \"kubernetes.io/projected/cbd4f0b9-e235-40f7-acef-68e1051fc918-kube-api-access-rxmkl\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.820856 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467bdac4-5015-4338-a6b1-c7591c441847-istiod-ca-cert\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.820877 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467bdac4-5015-4338-a6b1-c7591c441847-istio-podinfo\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.820892 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-envoy\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.820910 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-istio-data\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.820924 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-workload-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.820939 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-podinfo\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.820956 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-token\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.820978 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-workload-certs\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.821198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.821001 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467bdac4-5015-4338-a6b1-c7591c441847-istio-token\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.898624 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.898596 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" Oct 31 00:45:57.922020 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.921983 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxmkl\" (UniqueName: \"kubernetes.io/projected/cbd4f0b9-e235-40f7-acef-68e1051fc918-kube-api-access-rxmkl\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.922230 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922028 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467bdac4-5015-4338-a6b1-c7591c441847-istiod-ca-cert\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.922230 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922060 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467bdac4-5015-4338-a6b1-c7591c441847-istio-podinfo\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.922230 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922087 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-envoy\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.922230 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922111 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-istio-data\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.922230 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922138 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-workload-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.922506 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922260 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-podinfo\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.922506 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922288 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-token\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.922506 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922326 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-workload-certs\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.922506 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922365 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467bdac4-5015-4338-a6b1-c7591c441847-istio-token\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.922506 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922424 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-credential-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.922506 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922454 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-data\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.922867 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922567 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-workload-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.922867 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922579 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-workload-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.922867 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922670 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-workload-certs\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.922867 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922724 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-credential-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.922867 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922758 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-istio-envoy\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.922867 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922813 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd4f0b9-e235-40f7-acef-68e1051fc918-istiod-ca-cert\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.922867 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922844 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l947c\" (UniqueName: \"kubernetes.io/projected/467bdac4-5015-4338-a6b1-c7591c441847-kube-api-access-l947c\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.923220 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.922903 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467bdac4-5015-4338-a6b1-c7591c441847-istiod-ca-cert\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.923220 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.923185 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-workload-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.923332 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.923213 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-credential-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.923332 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.923315 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-workload-certs\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.923438 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.923414 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-credential-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.923649 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.923614 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-data\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.923728 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.923683 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-workload-certs\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.923942 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.923889 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-istio-data\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.924455 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.924090 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd4f0b9-e235-40f7-acef-68e1051fc918-istiod-ca-cert\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.925811 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.925735 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-podinfo\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.925936 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.925912 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-istio-envoy\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.926127 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.926110 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-envoy\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.927409 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.927379 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467bdac4-5015-4338-a6b1-c7591c441847-istio-podinfo\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.945216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.945184 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxmkl\" (UniqueName: \"kubernetes.io/projected/cbd4f0b9-e235-40f7-acef-68e1051fc918-kube-api-access-rxmkl\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.957595 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.957524 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l947c\" (UniqueName: \"kubernetes.io/projected/467bdac4-5015-4338-a6b1-c7591c441847-kube-api-access-l947c\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:57.957949 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.957927 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-token\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:57.958871 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:57.958846 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467bdac4-5015-4338-a6b1-c7591c441847-istio-token\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:58.004979 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.004855 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:58.023810 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.023776 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1283fbf-f92e-465e-99f6-4582a8022574-kube-api-access-xgpfm\" (UniqueName: \"kubernetes.io/projected/a1283fbf-f92e-465e-99f6-4582a8022574-kube-api-access-xgpfm\") pod \"a1283fbf-f92e-465e-99f6-4582a8022574\" (UID: \"a1283fbf-f92e-465e-99f6-4582a8022574\") " Oct 31 00:45:58.024304 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.023972 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1283fbf-f92e-465e-99f6-4582a8022574-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1283fbf-f92e-465e-99f6-4582a8022574-maas-api-xks-ca\") pod \"a1283fbf-f92e-465e-99f6-4582a8022574\" (UID: \"a1283fbf-f92e-465e-99f6-4582a8022574\") " Oct 31 00:45:58.024304 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.024068 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1283fbf-f92e-465e-99f6-4582a8022574-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1283fbf-f92e-465e-99f6-4582a8022574-maas-api-openshift-ca\") pod \"a1283fbf-f92e-465e-99f6-4582a8022574\" (UID: \"a1283fbf-f92e-465e-99f6-4582a8022574\") " Oct 31 00:45:58.025037 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.025006 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1283fbf-f92e-465e-99f6-4582a8022574-maas-api-openshift-ca" pod "a1283fbf-f92e-465e-99f6-4582a8022574" (UID: "a1283fbf-f92e-465e-99f6-4582a8022574"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:58.027376 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.027298 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1283fbf-f92e-465e-99f6-4582a8022574-kube-api-access-xgpfm" pod "a1283fbf-f92e-465e-99f6-4582a8022574" (UID: "a1283fbf-f92e-465e-99f6-4582a8022574"). InnerVolumeSpecName "kube-api-access-xgpfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:58.027755 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.027721 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1283fbf-f92e-465e-99f6-4582a8022574-maas-api-xks-ca" pod "a1283fbf-f92e-465e-99f6-4582a8022574" (UID: "a1283fbf-f92e-465e-99f6-4582a8022574"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:58.035848 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.035815 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh"] Oct 31 00:45:58.036400 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.036379 4519 cpu_manager.go:414] "containerMap: removing container" podUID="a1283fbf-f92e-465e-99f6-4582a8022574" containerName="revoke-keys" Oct 31 00:45:58.036400 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.036399 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1283fbf-f92e-465e-99f6-4582a8022574" containerName="revoke-keys" Oct 31 00:45:58.036545 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.036492 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1283fbf-f92e-465e-99f6-4582a8022574" containerName="revoke-keys" Oct 31 00:45:58.040907 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.040874 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.046242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.044442 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:58.062022 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.062000 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-7f1496-openshift-default-dockercfg-jpdcs\"" Oct 31 00:45:58.074679 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.074429 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh"] Oct 31 00:45:58.088656 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.084025 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g"] Oct 31 00:45:58.125812 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.125780 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1283fbf-f92e-465e-99f6-4582a8022574-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:58.126020 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.126001 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1283fbf-f92e-465e-99f6-4582a8022574-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:58.126126 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.126113 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgpfm\" (UniqueName: \"kubernetes.io/projected/a1283fbf-f92e-465e-99f6-4582a8022574-kube-api-access-xgpfm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:58.140692 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.140006 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.162300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.162279 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-1d4a7e-openshift-default-dockercfg-sn4k2\"" Oct 31 00:45:58.173608 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.173416 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g"] Oct 31 00:45:58.226872 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.226779 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5498f31e-f243-43b8-992a-56cbccdc239c-istiod-ca-cert\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.226872 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.226818 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5498f31e-f243-43b8-992a-56cbccdc239c-istio-podinfo\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.227101 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.226969 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-workload-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.227101 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.226991 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-istio-envoy\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.227101 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.227008 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-istio-data\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.227101 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.227072 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5clh4\" (UniqueName: \"kubernetes.io/projected/5498f31e-f243-43b8-992a-56cbccdc239c-kube-api-access-5clh4\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.227283 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.227122 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5498f31e-f243-43b8-992a-56cbccdc239c-istio-token\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.227283 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.227229 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-workload-certs\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.227283 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.227252 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-credential-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.328657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.328315 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-workload-certs\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.328657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.328398 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-workload-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.328657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.328429 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-istio-envoy\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.328657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.328457 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-istio-data\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.328657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.328493 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5clh4\" (UniqueName: \"kubernetes.io/projected/5498f31e-f243-43b8-992a-56cbccdc239c-kube-api-access-5clh4\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.328657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.328518 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ntpf\" (UniqueName: \"kubernetes.io/projected/f6b39483-2371-4570-843b-c4421cad360f-kube-api-access-2ntpf\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.328657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.328542 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5498f31e-f243-43b8-992a-56cbccdc239c-istio-token\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.328657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.328598 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b39483-2371-4570-843b-c4421cad360f-istio-token\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.328657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.328662 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-workload-socket\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.329248 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.328733 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b39483-2371-4570-843b-c4421cad360f-istiod-ca-cert\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.329248 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.328775 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-workload-certs\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.329248 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.328801 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-credential-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.329248 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.328859 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-credential-socket\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.329248 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.328887 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-istio-envoy\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.329248 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.328951 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5498f31e-f243-43b8-992a-56cbccdc239c-istiod-ca-cert\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.329248 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.328979 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5498f31e-f243-43b8-992a-56cbccdc239c-istio-podinfo\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.329248 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.329006 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-istio-data\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.329248 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.329060 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b39483-2371-4570-843b-c4421cad360f-istio-podinfo\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.329248 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.329121 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-istio-data\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.329782 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.329400 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-workload-certs\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.329782 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.329529 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-credential-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.336958 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.330279 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5498f31e-f243-43b8-992a-56cbccdc239c-istiod-ca-cert\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.336958 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.330531 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-workload-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.337522 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.337444 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-istio-envoy\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.345074 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.345011 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5498f31e-f243-43b8-992a-56cbccdc239c-istio-token\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.347552 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.347521 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5clh4\" (UniqueName: \"kubernetes.io/projected/5498f31e-f243-43b8-992a-56cbccdc239c-kube-api-access-5clh4\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.356211 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.353143 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5498f31e-f243-43b8-992a-56cbccdc239c-istio-podinfo\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.366316 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.363761 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:45:58.367853 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.367820 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6"] Oct 31 00:45:58.370263 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:45:58.370236 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd4f0b9_e235_40f7_acef_68e1051fc918.slice/crio-5c622d48770f27b2e181435fad3b6b89f9e8f130f904de541d63383cfe4f2f01 WatchSource:0}: Error finding container 5c622d48770f27b2e181435fad3b6b89f9e8f130f904de541d63383cfe4f2f01: Status 404 returned error can't find the container with id 5c622d48770f27b2e181435fad3b6b89f9e8f130f904de541d63383cfe4f2f01 Oct 31 00:45:58.375123 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.375083 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:58.375196 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.375173 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:58.375339 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.375214 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:58.389176 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.389145 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6"] Oct 31 00:45:58.390593 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:45:58.390555 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467bdac4_5015_4338_a6b1_c7591c441847.slice/crio-eab607a8926ad29d6f5e4f84d738f4a654d2c1e84e0b3aa9b9d170e845c4d124 WatchSource:0}: Error finding container eab607a8926ad29d6f5e4f84d738f4a654d2c1e84e0b3aa9b9d170e845c4d124: Status 404 returned error can't find the container with id eab607a8926ad29d6f5e4f84d738f4a654d2c1e84e0b3aa9b9d170e845c4d124 Oct 31 00:45:58.394242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.394206 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:58.394324 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.394303 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:58.394378 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.394358 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:58.430659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.430058 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ntpf\" (UniqueName: \"kubernetes.io/projected/f6b39483-2371-4570-843b-c4421cad360f-kube-api-access-2ntpf\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.430659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.430103 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b39483-2371-4570-843b-c4421cad360f-istio-token\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.430659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.430133 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-workload-socket\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.430659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.430175 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b39483-2371-4570-843b-c4421cad360f-istiod-ca-cert\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.430659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.430220 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-credential-socket\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.430659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.430244 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-istio-envoy\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.430659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.430283 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-istio-data\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.430659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.430318 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b39483-2371-4570-843b-c4421cad360f-istio-podinfo\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.430659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.430346 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-workload-certs\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.431275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.430708 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-workload-certs\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.431330 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.431281 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-workload-socket\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.431835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.431743 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b39483-2371-4570-843b-c4421cad360f-istiod-ca-cert\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.431953 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.431927 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-credential-socket\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.433010 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.432967 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-istio-data\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.436231 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.435674 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b39483-2371-4570-843b-c4421cad360f-istio-podinfo\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.459240 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.459207 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-istio-envoy\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.459405 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.459279 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ntpf\" (UniqueName: \"kubernetes.io/projected/f6b39483-2371-4570-843b-c4421cad360f-kube-api-access-2ntpf\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.464151 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.464122 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b39483-2371-4570-843b-c4421cad360f-istio-token\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.472618 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.472403 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:45:58.497139 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.497073 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:58.701045 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:45:58.700114 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5498f31e_f243_43b8_992a_56cbccdc239c.slice/crio-c5ca46aa5ac983bbcc7096115815db67ebdec49fa41010e55172d0bb9537289f WatchSource:0}: Error finding container c5ca46aa5ac983bbcc7096115815db67ebdec49fa41010e55172d0bb9537289f: Status 404 returned error can't find the container with id c5ca46aa5ac983bbcc7096115815db67ebdec49fa41010e55172d0bb9537289f Oct 31 00:45:58.704271 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.704241 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh"] Oct 31 00:45:58.704463 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.704256 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:58.704697 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.704619 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:58.704863 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.704842 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:58.765705 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.758026 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" event={"ID":"467bdac4-5015-4338-a6b1-c7591c441847","Type":"ContainerStarted","Data":"238f611f7c542b7063070f09d96c36047096debb888a7cee60a445f152d69b36"} Oct 31 00:45:58.765705 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.758089 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" event={"ID":"467bdac4-5015-4338-a6b1-c7591c441847","Type":"ContainerStarted","Data":"eab607a8926ad29d6f5e4f84d738f4a654d2c1e84e0b3aa9b9d170e845c4d124"} Oct 31 00:45:58.778440 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.776952 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" event={"ID":"a1283fbf-f92e-465e-99f6-4582a8022574","Type":"ContainerDied","Data":"5cbb0220cdf653da23a43315d7d2f150b5ca8eda5a0e38c70345b7415ab977b1"} Oct 31 00:45:58.778440 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.777007 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cbb0220cdf653da23a43315d7d2f150b5ca8eda5a0e38c70345b7415ab977b1" Oct 31 00:45:58.778440 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.776967 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd" Oct 31 00:45:58.781110 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.781055 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" podStartSLOduration=1.7810360809999999 podStartE2EDuration="1.781036081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:58.777547845 +0000 UTC m=+754.806728232" watchObservedRunningTime="2026-10-31 00:45:58.781036081 +0000 UTC m=+754.810216476" Oct 31 00:45:58.786365 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.786327 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" event={"ID":"5498f31e-f243-43b8-992a-56cbccdc239c","Type":"ContainerStarted","Data":"c5ca46aa5ac983bbcc7096115815db67ebdec49fa41010e55172d0bb9537289f"} Oct 31 00:45:58.788316 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.788288 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" event={"ID":"cbd4f0b9-e235-40f7-acef-68e1051fc918","Type":"ContainerStarted","Data":"bb022f8a4c6192c012196d3238742fb91f603287c22564bbff87399536c5ef29"} Oct 31 00:45:58.788515 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.788494 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" event={"ID":"cbd4f0b9-e235-40f7-acef-68e1051fc918","Type":"ContainerStarted","Data":"5c622d48770f27b2e181435fad3b6b89f9e8f130f904de541d63383cfe4f2f01"} Oct 31 00:45:58.802110 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.801955 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g"] Oct 31 00:45:58.827668 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:45:58.824546 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b39483_2371_4570_843b_c4421cad360f.slice/crio-b2e6208484d8799c5522bc1a8cac1707c664ad379b820f957e805f78226de834 WatchSource:0}: Error finding container b2e6208484d8799c5522bc1a8cac1707c664ad379b820f957e805f78226de834: Status 404 returned error can't find the container with id b2e6208484d8799c5522bc1a8cac1707c664ad379b820f957e805f78226de834 Oct 31 00:45:58.828693 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.828263 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:58.828693 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.828329 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:58.828693 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:58.828359 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:59.007903 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:59.007358 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:45:59.007903 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:59.007485 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:45:59.007903 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:59.007529 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" podUID="cbd4f0b9-e235-40f7-acef-68e1051fc918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 00:45:59.055911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:59.054692 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:45:59.055911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:59.054858 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:45:59.055911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:59.054902 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" podUID="467bdac4-5015-4338-a6b1-c7591c441847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:45:59.846659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:59.845994 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" podStartSLOduration=2.845970545 podStartE2EDuration="2.845970545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:58.808985801 +0000 UTC m=+754.838166190" watchObservedRunningTime="2026-10-31 00:45:59.845970545 +0000 UTC m=+755.875150940" Oct 31 00:45:59.870671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:59.870355 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:59.899660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:45:59.870987 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb"] Oct 31 00:46:00.008786 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.008747 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:46:00.012660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.009030 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" podUID="cbd4f0b9-e235-40f7-acef-68e1051fc918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 00:46:00.195728 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.023858 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb"] Oct 31 00:46:00.195728 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.194905 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" event={"ID":"f6b39483-2371-4570-843b-c4421cad360f","Type":"ContainerStarted","Data":"e6a7506007587f767fe365ecda819b3de4bf0d1cebc74989349a037681156d81"} Oct 31 00:46:00.195728 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.194950 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" event={"ID":"f6b39483-2371-4570-843b-c4421cad360f","Type":"ContainerStarted","Data":"b2e6208484d8799c5522bc1a8cac1707c664ad379b820f957e805f78226de834"} Oct 31 00:46:00.196070 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.195988 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" podStartSLOduration=2.195971257 podStartE2EDuration="2.195971257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:00.194951133 +0000 UTC m=+756.224131523" watchObservedRunningTime="2026-10-31 00:46:00.195971257 +0000 UTC m=+756.225151646" Oct 31 00:46:00.198046 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.023955 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.199919 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.057509 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:00.210961 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.063760 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:46:00.211240 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.211205 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" podUID="467bdac4-5015-4338-a6b1-c7591c441847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:46:00.221083 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.221049 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f99ac0\"/\"kube-root-ca.crt\"" Oct 31 00:46:00.221454 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.221435 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f99ac0\"/\"default-dockercfg-cpj7p\"" Oct 31 00:46:00.221690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.221674 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f99ac0\"/\"distinct-w0-f99ac0-kserve-self-signed-certs\"" Oct 31 00:46:00.221876 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.221863 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f99ac0\"/\"openshift-service-ca.crt\"" Oct 31 00:46:00.222670 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.107527 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-model-cache\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.224566 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.224543 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e27be156-a2bf-407a-a15c-b348f3e87105-tls-certs\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.224787 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.224769 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-tmp-dir\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.225164 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.225148 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-dshm\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.227501 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.227478 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-home\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.227788 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.227769 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ssq9\" (UniqueName: \"kubernetes.io/projected/e27be156-a2bf-407a-a15c-b348f3e87105-kube-api-access-7ssq9\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.263062 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.263000 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" podStartSLOduration=3.262982037 podStartE2EDuration="3.262982037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:00.245225993 +0000 UTC m=+756.274406383" watchObservedRunningTime="2026-10-31 00:46:00.262982037 +0000 UTC m=+756.292162428" Oct 31 00:46:00.286960 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.286909 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" event={"ID":"5498f31e-f243-43b8-992a-56cbccdc239c","Type":"ContainerStarted","Data":"bd61ba53498b29b792c2194cdbdd2024201a24521590e2e096d86e3479cf1cb9"} Oct 31 00:46:00.341661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.329922 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-dshm\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.342736 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.342702 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-home\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.343011 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.342992 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ssq9\" (UniqueName: \"kubernetes.io/projected/e27be156-a2bf-407a-a15c-b348f3e87105-kube-api-access-7ssq9\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.343137 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.343123 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-model-cache\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.343257 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.343244 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e27be156-a2bf-407a-a15c-b348f3e87105-tls-certs\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.343422 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.343406 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-tmp-dir\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.343953 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.343933 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-tmp-dir\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.344329 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.344309 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-home\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.344991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.344970 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-model-cache\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.349094 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.349069 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e27be156-a2bf-407a-a15c-b348f3e87105-tls-certs\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.356657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.350957 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-dshm\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.360524 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.360494 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ssq9\" (UniqueName: \"kubernetes.io/projected/e27be156-a2bf-407a-a15c-b348f3e87105-kube-api-access-7ssq9\") pod \"distinct-w0-f99ac0-kserve-77f68668c7-4l8mb\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.369051 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.369019 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:46:00.376825 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.376457 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:46:00.376825 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.376516 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" podUID="5498f31e-f243-43b8-992a-56cbccdc239c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:46:00.477818 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.477710 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:46:00.481809 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.481768 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:46:00.482061 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.482034 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" podUID="f6b39483-2371-4570-843b-c4421cad360f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:46:00.570921 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.570875 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:00.862660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.858689 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:00.930838 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.930788 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:46:00.976138 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:00.976074 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:01.032259 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.032206 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:46:01.032548 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.032516 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" podUID="cbd4f0b9-e235-40f7-acef-68e1051fc918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 00:46:01.048660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.045545 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:46:01.052786 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.052709 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" podUID="467bdac4-5015-4338-a6b1-c7591c441847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:46:01.080815 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.080765 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:01.368093 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.368054 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:46:01.368383 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.368352 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" podUID="5498f31e-f243-43b8-992a-56cbccdc239c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:46:01.500657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.476053 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:46:01.577392 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.557217 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb"] Oct 31 00:46:01.577719 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.568346 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:01.578007 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.577967 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" podUID="f6b39483-2371-4570-843b-c4421cad360f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:46:01.676317 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.659890 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:01.682894 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.681770 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:01.820064 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.820018 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:01.861850 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.861807 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:01.862836 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.862808 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:46:01.948949 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.948844 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:46:01.950827 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.949782 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:46:01.981663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:01.977676 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:02.015647 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.015589 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:46:02.037149 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.036736 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:46:02.037149 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.036793 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" podUID="cbd4f0b9-e235-40f7-acef-68e1051fc918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 00:46:02.256123 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.256087 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:46:02.299783 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.299728 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" podUID="467bdac4-5015-4338-a6b1-c7591c441847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:46:02.341897 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.300507 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:02.371052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.371013 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:46:02.371367 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.371343 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" podUID="5498f31e-f243-43b8-992a-56cbccdc239c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:46:02.407732 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.407703 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:46:02.450263 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.448958 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:02.462303 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.462265 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" event={"ID":"e27be156-a2bf-407a-a15c-b348f3e87105","Type":"ContainerStarted","Data":"69ce29f94d578e274a4b54c1d20ddfdc44537ea5d667390a2f9aa14b349a320e"} Oct 31 00:46:02.484362 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.484332 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:46:02.486244 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.484869 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:46:02.486438 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.486415 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" podUID="f6b39483-2371-4570-843b-c4421cad360f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:46:02.689189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.689094 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:02.803681 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.725682 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:02.889656 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.839987 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:02.889656 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:02.840321 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:46:03.043536 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:03.042769 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:46:03.043536 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:03.042831 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" podUID="cbd4f0b9-e235-40f7-acef-68e1051fc918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 00:46:03.043536 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:03.042933 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:03.043918 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:03.042716 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:03.045940 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:03.044076 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:46:03.045940 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:03.044443 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:46:03.045940 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:03.044497 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:03.046270 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:03.046249 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:46:03.046408 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:03.046389 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" podUID="467bdac4-5015-4338-a6b1-c7591c441847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:46:03.417794 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:03.417698 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:46:03.451773 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:03.451720 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" podUID="5498f31e-f243-43b8-992a-56cbccdc239c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:46:03.536833 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:03.536788 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:46:03.540805 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:03.540736 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" podUID="f6b39483-2371-4570-843b-c4421cad360f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:46:03.554850 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:03.554793 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:46:03.608992 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:03.608949 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:03.842965 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:03.841744 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:04.014002 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.013964 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:46:04.014549 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.014517 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" podUID="cbd4f0b9-e235-40f7-acef-68e1051fc918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 00:46:04.061024 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.060978 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:04.061660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.061606 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:46:04.064798 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.064773 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:46:04.064944 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.064924 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" podUID="467bdac4-5015-4338-a6b1-c7591c441847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:46:04.065416 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.065392 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 00:46:04.066244 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.066219 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:04.066668 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.066650 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:46:04.106784 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.106677 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:04.376729 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.370086 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:46:04.378479 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.378448 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" podUID="5498f31e-f243-43b8-992a-56cbccdc239c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:46:04.383104 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.378346 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:04.404980 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.404951 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:46:04.422674 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.420960 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:46:04.475100 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.475059 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:46:04.475371 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.475339 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" podUID="f6b39483-2371-4570-843b-c4421cad360f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:46:04.723741 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.682890 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded" Oct 31 00:46:04.855660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:04.851721 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:05.009493 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:05.009456 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:46:05.009801 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:05.009773 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" podUID="cbd4f0b9-e235-40f7-acef-68e1051fc918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 00:46:05.046317 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:05.046274 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:46:05.046990 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:05.046943 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" podUID="467bdac4-5015-4338-a6b1-c7591c441847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:46:05.055551 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:05.055520 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:46:05.066944 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:05.066897 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:05.243972 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:05.243936 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:46:05.427726 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:05.423589 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:46:05.428004 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:05.427966 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" podUID="5498f31e-f243-43b8-992a-56cbccdc239c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:46:05.432784 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:05.432751 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:46:05.433292 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:05.433272 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:46:05.467661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:05.436349 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:05.524728 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:05.520084 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:05.576664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:05.520935 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:46:05.576664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:05.574048 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" podUID="f6b39483-2371-4570-843b-c4421cad360f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:46:05.576664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:05.526816 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:05.886716 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:05.862754 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:06.017328 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:06.017002 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:46:06.017328 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:06.017066 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" podUID="cbd4f0b9-e235-40f7-acef-68e1051fc918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 00:46:06.065660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:06.064716 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:06.073664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:06.071775 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:46:06.073664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:06.071832 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" podUID="467bdac4-5015-4338-a6b1-c7591c441847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:46:06.091107 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:06.091059 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:06.333963 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:06.332936 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:46:06.371672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:06.370438 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:46:06.371672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:06.370544 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" podUID="5498f31e-f243-43b8-992a-56cbccdc239c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:46:06.444661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:06.436791 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:06.474173 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:06.474121 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:46:06.474602 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:06.474463 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" podUID="f6b39483-2371-4570-843b-c4421cad360f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:46:06.523500 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:06.523431 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:46:06.524058 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:06.524022 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:46:06.653729 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:06.637692 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:46:06.804858 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:06.804064 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:46:06.878988 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:06.878344 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" event={"ID":"e27be156-a2bf-407a-a15c-b348f3e87105","Type":"ContainerStarted","Data":"a80a3f83d9ccc01363f66ec7f36548b74dbcc02c7cf5c91c0fcac065f45a700f"} Oct 31 00:46:07.063661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:07.057386 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:46:07.063661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:07.057445 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" podUID="467bdac4-5015-4338-a6b1-c7591c441847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:46:07.178667 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:07.172776 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:07.408878 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:07.408761 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:46:07.409757 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:07.409735 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:46:07.414228 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:07.413842 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:46:07.441684 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:07.440358 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podStartSLOduration=8.440339078 podStartE2EDuration="8.440339078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:06.903261497 +0000 UTC m=+762.932441888" watchObservedRunningTime="2026-10-31 00:46:07.440339078 +0000 UTC m=+763.469519468" Oct 31 00:46:07.475511 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:07.475478 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:46:07.475774 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:07.475747 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" podUID="f6b39483-2371-4570-843b-c4421cad360f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:46:07.577723 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:07.574889 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:07.716829 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:07.716719 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:46:07.717700 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:07.717675 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:46:07.720292 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:07.720270 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:46:08.210882 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:08.210850 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:46:08.211589 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:08.211568 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:46:08.212400 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:08.212379 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:46:08.478520 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:08.478443 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:46:08.478874 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:08.478853 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:46:08.479543 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:08.479524 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:46:08.700005 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:08.699977 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:09.523323 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:09.523290 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:10.037662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:10.036190 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:10.633660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:10.633207 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:10.777608 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:10.777571 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:11.012095 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:11.012062 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:11.708554 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:11.707490 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:12.303787 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:12.303752 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:13.260132 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:13.260103 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:14.130447 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:14.130410 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:14.856659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:14.856597 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:15.326254 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.326213 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp"] Oct 31 00:46:15.330955 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.330919 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.338487 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.338456 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-44bc67\"/\"distinct-w5-44bc67-kserve-self-signed-certs\"" Oct 31 00:46:15.340605 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.340560 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp"] Oct 31 00:46:15.356705 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.356666 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:15.427819 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.427776 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-dshm\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.428023 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.427850 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-tmp-dir\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.428023 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.427967 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-home\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.428153 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.428127 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndptw\" (UniqueName: \"kubernetes.io/projected/afa77fbd-2015-4986-bf98-f4858d87f0bf-kube-api-access-ndptw\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.428194 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.428180 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa77fbd-2015-4986-bf98-f4858d87f0bf-tls-certs\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.428318 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.428301 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-model-cache\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.535517 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.535468 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa77fbd-2015-4986-bf98-f4858d87f0bf-tls-certs\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.536539 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.536507 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-model-cache\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.537504 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.537485 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-dshm\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.538039 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.538018 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-tmp-dir\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.538650 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.538618 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-home\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.538993 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.538975 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndptw\" (UniqueName: \"kubernetes.io/projected/afa77fbd-2015-4986-bf98-f4858d87f0bf-kube-api-access-ndptw\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.540985 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.540961 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-home\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.541142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.537122 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-model-cache\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.541230 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.538531 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-tmp-dir\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.547040 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.547006 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-dshm\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.547952 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.547923 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa77fbd-2015-4986-bf98-f4858d87f0bf-tls-certs\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.576714 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.573877 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndptw\" (UniqueName: \"kubernetes.io/projected/afa77fbd-2015-4986-bf98-f4858d87f0bf-kube-api-access-ndptw\") pod \"distinct-w5-44bc67-kserve-59979f5cd8-knbmp\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:15.619927 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.619886 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:15.649356 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:15.649018 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:16.010495 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:16.010465 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp"] Oct 31 00:46:16.011810 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:46:16.011775 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafa77fbd_2015_4986_bf98_f4858d87f0bf.slice/crio-d39da3c482e8576f1b75bcaba42ffe7b4a769c332c36558e04fd060b0f8db62b WatchSource:0}: Error finding container d39da3c482e8576f1b75bcaba42ffe7b4a769c332c36558e04fd060b0f8db62b: Status 404 returned error can't find the container with id d39da3c482e8576f1b75bcaba42ffe7b4a769c332c36558e04fd060b0f8db62b Oct 31 00:46:16.390504 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:16.390399 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:16.972128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:16.972089 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" event={"ID":"afa77fbd-2015-4986-bf98-f4858d87f0bf","Type":"ContainerStarted","Data":"467ba0efe527b424c9260b979ab129c36f1f13b277c717465224c584ed0f4cc8"} Oct 31 00:46:16.972333 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:16.972137 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" event={"ID":"afa77fbd-2015-4986-bf98-f4858d87f0bf","Type":"ContainerStarted","Data":"d39da3c482e8576f1b75bcaba42ffe7b4a769c332c36558e04fd060b0f8db62b"} Oct 31 00:46:16.991451 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:16.991389 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podStartSLOduration=1.99137425 podStartE2EDuration="1.99137425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:16.98913976 +0000 UTC m=+773.018320178" watchObservedRunningTime="2026-10-31 00:46:16.99137425 +0000 UTC m=+773.020554640" Oct 31 00:46:17.061091 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:17.061037 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:17.543573 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:46:17.543536 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:46:17.650381 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:17.650350 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:17.744565 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:17.744531 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:18.443762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:18.443727 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:19.048857 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:19.048823 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:19.753999 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:19.753965 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:20.434509 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:20.434474 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:21.152629 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:21.152591 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:21.838887 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:21.838851 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:22.452521 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:22.452489 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:23.141509 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:23.141470 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:23.468792 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:23.468710 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:23.892764 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:23.892727 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:24.650458 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:24.650425 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:25.337926 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:25.337894 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:25.650272 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:25.650192 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:25.663234 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:25.663203 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:26.035824 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:26.035787 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:26.044613 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:26.044584 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:26.659081 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:26.659049 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:27.022924 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.022885 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:27.535667 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.535623 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:27.633147 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.633108 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz"] Oct 31 00:46:27.640208 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.640175 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.706698 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.689046 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz"] Oct 31 00:46:27.796603 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.795246 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393b6b08-821f-4788-9da8-230f14d4e867-istio-token\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.796903 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.796869 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393b6b08-821f-4788-9da8-230f14d4e867-istiod-ca-cert\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.797113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.797094 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d6qm\" (UniqueName: \"kubernetes.io/projected/393b6b08-821f-4788-9da8-230f14d4e867-kube-api-access-5d6qm\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.797277 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.797252 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-workload-certs\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.797429 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.797406 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393b6b08-821f-4788-9da8-230f14d4e867-istio-podinfo\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.797611 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.797596 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-istio-data\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.797854 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.797827 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-credential-socket\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.797986 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.797967 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-workload-socket\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.798067 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.798032 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-istio-envoy\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.899780 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.899732 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-istio-envoy\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.901380 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.901353 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393b6b08-821f-4788-9da8-230f14d4e867-istio-token\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.901582 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.901388 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393b6b08-821f-4788-9da8-230f14d4e867-istiod-ca-cert\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.901582 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.901466 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d6qm\" (UniqueName: \"kubernetes.io/projected/393b6b08-821f-4788-9da8-230f14d4e867-kube-api-access-5d6qm\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.901582 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.901523 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-workload-certs\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.901582 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.901573 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393b6b08-821f-4788-9da8-230f14d4e867-istio-podinfo\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.901911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.901668 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-istio-data\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.901911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.901703 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-credential-socket\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.901911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.901812 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-workload-socket\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.902239 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.902218 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-workload-socket\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.902469 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.902453 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-workload-certs\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.902851 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.902820 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393b6b08-821f-4788-9da8-230f14d4e867-istiod-ca-cert\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.903114 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.903090 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-istio-data\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.903613 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.903319 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-credential-socket\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.905415 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.905394 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393b6b08-821f-4788-9da8-230f14d4e867-istio-podinfo\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.911398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.911345 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393b6b08-821f-4788-9da8-230f14d4e867-istio-token\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.914201 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.914173 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d6qm\" (UniqueName: \"kubernetes.io/projected/393b6b08-821f-4788-9da8-230f14d4e867-kube-api-access-5d6qm\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:27.915177 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:27.915148 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-istio-envoy\") pod \"e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:28.104718 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.103838 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:28.179727 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.177263 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq"] Oct 31 00:46:28.183609 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.183580 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.196902 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.196865 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq"] Oct 31 00:46:28.370503 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.370402 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-credential-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.370942 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.370914 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-token\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.371142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.371127 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-podinfo\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.379123 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.371293 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-data\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.379413 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.379376 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-envoy\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.379714 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.379693 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-workload-certs\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.383745 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.383714 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brgmx\" (UniqueName: \"kubernetes.io/projected/4edab197-6a3f-457a-acab-82d4cebc7b0f-kube-api-access-brgmx\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.396953 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.396921 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-workload-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.397210 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.397189 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4edab197-6a3f-457a-acab-82d4cebc7b0f-istiod-ca-cert\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.499196 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.499143 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-token\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.499537 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.499516 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-podinfo\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.499743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.499729 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-data\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.499878 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.499865 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-envoy\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.500043 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.500010 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-workload-certs\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.500190 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.500170 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brgmx\" (UniqueName: \"kubernetes.io/projected/4edab197-6a3f-457a-acab-82d4cebc7b0f-kube-api-access-brgmx\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.500488 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.500472 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-workload-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.500664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.500626 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4edab197-6a3f-457a-acab-82d4cebc7b0f-istiod-ca-cert\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.523680 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.523588 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-credential-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.523912 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.505147 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-workload-certs\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.524057 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.505376 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-data\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.524197 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.524170 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-credential-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.524462 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.518272 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-podinfo\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.524587 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.518333 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brgmx\" (UniqueName: \"kubernetes.io/projected/4edab197-6a3f-457a-acab-82d4cebc7b0f-kube-api-access-brgmx\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.524717 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.518956 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4edab197-6a3f-457a-acab-82d4cebc7b0f-istiod-ca-cert\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.524828 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.515120 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-envoy\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.524931 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.504430 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-workload-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.539433 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.539396 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-token\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:28.820739 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:28.819877 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:29.334716 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:29.323282 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz"] Oct 31 00:46:29.414663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:29.387435 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:29.414663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:29.413438 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:29.414663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:29.413526 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:29.794371 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:29.793006 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq"] Oct 31 00:46:29.814659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:29.810753 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8"] Oct 31 00:46:29.878252 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:29.859940 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:29.878252 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:29.861541 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8"] Oct 31 00:46:29.921845 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:29.917208 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-tmp-dir\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:29.921845 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:29.917291 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-home\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:29.921845 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:29.917328 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-model-cache\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:29.921845 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:29.917368 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-dshm\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:29.921845 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:29.917407 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnfrr\" (UniqueName: \"kubernetes.io/projected/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-kube-api-access-nnfrr\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:29.921845 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:29.917470 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-tls-certs\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:29.933336 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:29.929484 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:29.935418 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:29.935380 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b433e6\"/\"distinct-w2-b433e6-kserve-self-signed-certs\"" Oct 31 00:46:30.014112 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.014064 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:30.014437 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.014411 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:30.014721 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.014684 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:30.037533 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.036526 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-tmp-dir\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:30.037533 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.036630 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-home\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:30.037533 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.036695 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-model-cache\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:30.037533 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.036741 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-dshm\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:30.037533 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.036786 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnfrr\" (UniqueName: \"kubernetes.io/projected/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-kube-api-access-nnfrr\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:30.037533 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.036859 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-tls-certs\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:30.057460 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.057423 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:30.068692 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.059369 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-model-cache\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:30.068692 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.059683 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-tmp-dir\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:30.068692 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.059972 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-home\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:30.421033 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.420990 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" event={"ID":"393b6b08-821f-4788-9da8-230f14d4e867","Type":"ContainerStarted","Data":"11929329d7d3a797dbf1b9bcbe81de40f92972aa62858c5087ef6f0f6984d00b"} Oct 31 00:46:30.492906 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.492862 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" event={"ID":"393b6b08-821f-4788-9da8-230f14d4e867","Type":"ContainerStarted","Data":"c7c7ba4ed1f3444af753ddeb3ebe6472e1655bc1f5cff105595e5d6d99b26cb4"} Oct 31 00:46:30.493397 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.441803 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-dshm\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:30.493581 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.437601 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:30.494872 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.494096 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" event={"ID":"4edab197-6a3f-457a-acab-82d4cebc7b0f","Type":"ContainerStarted","Data":"8b26dfb6d98df8cd5ae310489885c2a52671560af6289f54d36f68027dcb3c18"} Oct 31 00:46:30.494872 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.494810 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-tls-certs\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:30.495773 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.495750 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnfrr\" (UniqueName: \"kubernetes.io/projected/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-kube-api-access-nnfrr\") pod \"distinct-w2-b433e6-kserve-7656d7bb48-xcfh8\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:30.515133 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.515068 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" podStartSLOduration=3.515047322 podStartE2EDuration="3.515047322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:30.494622344 +0000 UTC m=+786.523802771" watchObservedRunningTime="2026-10-31 00:46:30.515047322 +0000 UTC m=+786.544227712" Oct 31 00:46:30.592357 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:30.592323 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:31.104719 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:31.104685 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:31.106684 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:31.106646 4519 patch_prober.go:29] interesting pod/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:46:31.106855 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:31.106708 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" podUID="393b6b08-821f-4788-9da8-230f14d4e867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:46:31.123071 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:31.123037 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8"] Oct 31 00:46:31.123771 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:46:31.123745 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod193c8acd_7eb4_49f3_b2b3_0c398cdd9c02.slice/crio-f9502898d7bae49ef1fccc67011fcdb8014f625c919a509d548359b926988d4f WatchSource:0}: Error finding container f9502898d7bae49ef1fccc67011fcdb8014f625c919a509d548359b926988d4f: Status 404 returned error can't find the container with id f9502898d7bae49ef1fccc67011fcdb8014f625c919a509d548359b926988d4f Oct 31 00:46:31.311510 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:31.311468 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:31.501975 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:31.501936 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" event={"ID":"4edab197-6a3f-457a-acab-82d4cebc7b0f","Type":"ContainerStarted","Data":"6348482c530ea8af1d77c1fa72d36d3c3a545c2f1d3c44960310b79809c95142"} Oct 31 00:46:31.503646 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:31.503602 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" event={"ID":"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02","Type":"ContainerStarted","Data":"73ef214360d8f16a916d2226135eb5455b1d62b7336fc626ee6ae391624265bb"} Oct 31 00:46:31.503791 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:31.503655 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" event={"ID":"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02","Type":"ContainerStarted","Data":"f9502898d7bae49ef1fccc67011fcdb8014f625c919a509d548359b926988d4f"} Oct 31 00:46:31.527965 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:31.527875 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" podStartSLOduration=3.527858902 podStartE2EDuration="3.527858902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:31.524176479 +0000 UTC m=+787.553356871" watchObservedRunningTime="2026-10-31 00:46:31.527858902 +0000 UTC m=+787.557039292" Oct 31 00:46:31.547661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:31.546486 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podStartSLOduration=2.546465224 podStartE2EDuration="2.546465224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:31.542072 +0000 UTC m=+787.571252390" watchObservedRunningTime="2026-10-31 00:46:31.546465224 +0000 UTC m=+787.575645616" Oct 31 00:46:31.597658 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:31.597483 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:31.821216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:31.821128 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:31.838659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:31.832823 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:46:31.838659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:31.832903 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" podUID="4edab197-6a3f-457a-acab-82d4cebc7b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:46:32.104712 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:32.104622 4519 patch_prober.go:29] interesting pod/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:46:32.104926 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:32.104906 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" podUID="393b6b08-821f-4788-9da8-230f14d4e867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:46:32.181784 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:32.181750 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:32.821417 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:32.821376 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:46:32.821943 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:32.821439 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" podUID="4edab197-6a3f-457a-acab-82d4cebc7b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:46:33.105102 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:33.105000 4519 patch_prober.go:29] interesting pod/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:46:33.105102 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:33.105058 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" podUID="393b6b08-821f-4788-9da8-230f14d4e867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:46:33.220569 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:33.220534 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:33.821267 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:33.821230 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:46:33.821448 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:33.821280 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" podUID="4edab197-6a3f-457a-acab-82d4cebc7b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:46:34.104529 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:34.104448 4519 patch_prober.go:29] interesting pod/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:46:34.104529 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:34.104493 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" podUID="393b6b08-821f-4788-9da8-230f14d4e867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:46:34.207810 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:34.207778 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:34.843662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:34.837771 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:46:34.843662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:34.837837 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" podUID="4edab197-6a3f-457a-acab-82d4cebc7b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:46:35.104531 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:35.104447 4519 patch_prober.go:29] interesting pod/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:46:35.104531 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:35.104507 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" podUID="393b6b08-821f-4788-9da8-230f14d4e867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:46:35.181787 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:35.181750 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:35.821042 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:35.820992 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:46:35.821264 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:35.821068 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" podUID="4edab197-6a3f-457a-acab-82d4cebc7b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:46:36.104532 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:36.104446 4519 patch_prober.go:29] interesting pod/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:46:36.104532 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:36.104492 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" podUID="393b6b08-821f-4788-9da8-230f14d4e867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:46:36.291033 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:36.291003 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:36.821283 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:36.821248 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:46:36.821590 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:36.821555 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" podUID="4edab197-6a3f-457a-acab-82d4cebc7b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:46:37.104763 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:37.104581 4519 patch_prober.go:29] interesting pod/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:46:37.104763 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:37.104654 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" podUID="393b6b08-821f-4788-9da8-230f14d4e867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:46:37.249996 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:37.249958 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:37.820654 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:37.820613 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:46:37.820820 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:37.820705 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" podUID="4edab197-6a3f-457a-acab-82d4cebc7b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:46:38.014329 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:38.014295 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:38.110715 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:38.105883 4519 patch_prober.go:29] interesting pod/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:46:38.110715 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:38.105942 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" podUID="393b6b08-821f-4788-9da8-230f14d4e867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:46:38.823110 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:38.823070 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:38.826513 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:38.826484 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:38.826975 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:38.826956 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:38.827755 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:38.827733 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:46:39.109877 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:39.109802 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:39.110186 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:39.110164 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:39.110978 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:39.110960 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:46:39.620602 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:39.620569 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:40.340702 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:40.340667 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:40.595881 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:40.595805 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:40.608413 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:40.608384 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:41.030652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:41.030608 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:41.608420 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:41.608393 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:41.721127 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:41.721096 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:42.436228 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:42.436201 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:42.771254 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:42.771215 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6"] Oct 31 00:46:42.780784 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:42.775930 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:42.794936 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:42.794899 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6"] Oct 31 00:46:42.958337 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:42.958305 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-token\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:42.958337 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:42.958341 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-podinfo\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:42.958605 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:42.958362 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk6zq\" (UniqueName: \"kubernetes.io/projected/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-kube-api-access-kk6zq\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:42.958605 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:42.958427 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-envoy\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:42.958605 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:42.958513 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-workload-certs\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:42.958605 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:42.958552 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-workload-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:42.958605 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:42.958578 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istiod-ca-cert\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:42.958864 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:42.958689 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-data\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:42.958864 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:42.958799 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-credential-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:42.985907 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:42.985868 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn"] Oct 31 00:46:42.989829 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:42.989809 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.030998 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.030918 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k"] Oct 31 00:46:43.035750 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.035712 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn"] Oct 31 00:46:43.035912 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.035844 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.054106 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.054077 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k"] Oct 31 00:46:43.059742 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.059712 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istiod-ca-cert\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.059949 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.059929 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-data\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.060089 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.060064 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-credential-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.060216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.060188 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-token\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.060273 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.060263 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-podinfo\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.060332 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.060288 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk6zq\" (UniqueName: \"kubernetes.io/projected/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-kube-api-access-kk6zq\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.060332 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.060324 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-envoy\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.060437 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.060409 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-workload-certs\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.060486 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.060464 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-workload-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.064652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.060929 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-workload-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.064652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.061483 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istiod-ca-cert\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.064652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.061792 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-data\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.064652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.062029 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-credential-socket\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.064652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.063530 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-workload-certs\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.068000 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.067979 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-podinfo\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.070938 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.070916 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-envoy\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.080056 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.080030 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk6zq\" (UniqueName: \"kubernetes.io/projected/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-kube-api-access-kk6zq\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.082085 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.082058 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-token\") pod \"e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.107373 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.107333 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162350 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-istio-envoy\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162400 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km4bt\" (UniqueName: \"kubernetes.io/projected/b22ec283-637b-4dd0-add9-76777f64e77c-kube-api-access-km4bt\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162469 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-istio-data\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162501 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-credential-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162524 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istiod-ca-cert\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162554 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvw9v\" (UniqueName: \"kubernetes.io/projected/6a11f261-4cc7-4ca9-8b5f-8920883108b4-kube-api-access-jvw9v\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162577 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-credential-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162676 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22ec283-637b-4dd0-add9-76777f64e77c-istio-podinfo\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162700 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-workload-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162720 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22ec283-637b-4dd0-add9-76777f64e77c-istio-token\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162756 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22ec283-637b-4dd0-add9-76777f64e77c-istiod-ca-cert\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162777 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-data\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162822 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-podinfo\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162854 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-envoy\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162877 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-workload-certs\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162903 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-workload-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162935 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-token\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.163275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.162967 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-workload-certs\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.184812 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.184770 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw"] Oct 31 00:46:43.190306 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.190281 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.205478 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.204629 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw"] Oct 31 00:46:43.264614 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.264581 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-workload-certs\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.264800 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.264689 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-istio-envoy\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.264800 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.264713 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km4bt\" (UniqueName: \"kubernetes.io/projected/b22ec283-637b-4dd0-add9-76777f64e77c-kube-api-access-km4bt\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.264800 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.264758 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-istio-data\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.264800 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.264779 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-credential-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.264800 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.264796 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istiod-ca-cert\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.265098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.264820 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvw9v\" (UniqueName: \"kubernetes.io/projected/6a11f261-4cc7-4ca9-8b5f-8920883108b4-kube-api-access-jvw9v\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.265098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.264843 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-credential-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.265098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.264900 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22ec283-637b-4dd0-add9-76777f64e77c-istio-podinfo\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.265098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.264921 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-workload-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.265098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.264937 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22ec283-637b-4dd0-add9-76777f64e77c-istio-token\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.265098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.264965 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22ec283-637b-4dd0-add9-76777f64e77c-istiod-ca-cert\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.265098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.264984 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-data\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.265098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.265011 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-podinfo\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.265098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.265031 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-envoy\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.265098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.265053 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-workload-certs\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.265098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.265076 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-workload-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.265098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.265079 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-workload-certs\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.265098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.265097 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-token\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.265734 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.265583 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-workload-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.265984 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.265958 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-credential-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.266238 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.266218 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-istio-data\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.266379 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.266357 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istiod-ca-cert\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.266519 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.266501 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22ec283-637b-4dd0-add9-76777f64e77c-istiod-ca-cert\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.266675 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.266646 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-workload-certs\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.266817 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.266798 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-workload-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.266956 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.266935 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-credential-socket\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.267056 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.267038 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-data\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.268974 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.268951 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22ec283-637b-4dd0-add9-76777f64e77c-istio-podinfo\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.269194 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.269177 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-envoy\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.269395 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.269377 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-istio-envoy\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.269765 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.269746 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-podinfo\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.279294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.279261 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-token\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.281588 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.281523 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km4bt\" (UniqueName: \"kubernetes.io/projected/b22ec283-637b-4dd0-add9-76777f64e77c-kube-api-access-km4bt\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.282055 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.282022 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvw9v\" (UniqueName: \"kubernetes.io/projected/6a11f261-4cc7-4ca9-8b5f-8920883108b4-kube-api-access-jvw9v\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.282149 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.282065 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22ec283-637b-4dd0-add9-76777f64e77c-istio-token\") pod \"e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.303057 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.303029 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:43.360277 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.359809 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:43.369796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.367065 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-istio-envoy\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.369796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.367186 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-workload-socket\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.369796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.367242 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-istio-data\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.369796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.367287 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzvd5\" (UniqueName: \"kubernetes.io/projected/2ec76a17-5431-4756-a824-862c354595c8-kube-api-access-jzvd5\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.369796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.367316 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-workload-certs\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.369796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.367339 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec76a17-5431-4756-a824-862c354595c8-istio-token\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.369796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.367400 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec76a17-5431-4756-a824-862c354595c8-istiod-ca-cert\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.369796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.367552 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-credential-socket\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.369796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.368220 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec76a17-5431-4756-a824-862c354595c8-istio-podinfo\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.376999 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.374585 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:43.501452 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.501418 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-workload-socket\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.501715 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.501486 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-istio-data\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.501715 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.501526 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzvd5\" (UniqueName: \"kubernetes.io/projected/2ec76a17-5431-4756-a824-862c354595c8-kube-api-access-jzvd5\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.501715 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.501550 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-workload-certs\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.501715 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.501574 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec76a17-5431-4756-a824-862c354595c8-istio-token\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.501715 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.501626 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec76a17-5431-4756-a824-862c354595c8-istiod-ca-cert\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.503879 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.502443 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-workload-socket\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.503879 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.502875 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-workload-certs\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.535141 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.529384 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-istio-data\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.535141 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.529611 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-credential-socket\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.535141 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.529705 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec76a17-5431-4756-a824-862c354595c8-istio-podinfo\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.535141 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.529751 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-istio-envoy\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.535141 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.529857 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec76a17-5431-4756-a824-862c354595c8-istiod-ca-cert\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.535141 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.529977 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-credential-socket\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.535141 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.530520 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6"] Oct 31 00:46:43.535141 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.530832 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzvd5\" (UniqueName: \"kubernetes.io/projected/2ec76a17-5431-4756-a824-862c354595c8-kube-api-access-jzvd5\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.535141 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:46:43.531404 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3eab170_3ff7_48a5_8f32_9acfcb6db28b.slice/crio-bade3321deff64910dd8f9cc66c529bdbd25e429761d03c2aed460ee1a01ee0b WatchSource:0}: Error finding container bade3321deff64910dd8f9cc66c529bdbd25e429761d03c2aed460ee1a01ee0b: Status 404 returned error can't find the container with id bade3321deff64910dd8f9cc66c529bdbd25e429761d03c2aed460ee1a01ee0b Oct 31 00:46:43.535141 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.534253 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec76a17-5431-4756-a824-862c354595c8-istio-podinfo\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.535141 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.534244 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:43.535141 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.534351 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:43.535141 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.534408 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:43.549475 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.549122 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-istio-envoy\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.562612 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.562386 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec76a17-5431-4756-a824-862c354595c8-istio-token\") pod \"e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.610333 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.610296 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn"] Oct 31 00:46:43.620678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.620598 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:43.620804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.620733 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:43.620804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.620781 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:43.631714 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.631045 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" event={"ID":"b22ec283-637b-4dd0-add9-76777f64e77c","Type":"ContainerStarted","Data":"c8ca2caad342673a22cbda2f2150f38d47c13fe2f5153320f5d158ccbeca2260"} Oct 31 00:46:43.633680 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.633514 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" event={"ID":"a3eab170-3ff7-48a5-8f32-9acfcb6db28b","Type":"ContainerStarted","Data":"bade3321deff64910dd8f9cc66c529bdbd25e429761d03c2aed460ee1a01ee0b"} Oct 31 00:46:43.783830 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.783798 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k"] Oct 31 00:46:43.785082 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:46:43.785049 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a11f261_4cc7_4ca9_8b5f_8920883108b4.slice/crio-046eba82f2e738d8e1b6e122515eb314b0305f5d2f384654466f146218eab092 WatchSource:0}: Error finding container 046eba82f2e738d8e1b6e122515eb314b0305f5d2f384654466f146218eab092: Status 404 returned error can't find the container with id 046eba82f2e738d8e1b6e122515eb314b0305f5d2f384654466f146218eab092 Oct 31 00:46:43.787835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.787796 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:43.787932 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.787898 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:43.787973 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.787944 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:43.824774 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.824742 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:43.913764 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:43.912996 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:44.011575 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:44.011494 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:44.168974 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:44.168941 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw"] Oct 31 00:46:44.175320 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:44.175283 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:44.175441 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:44.175362 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:44.175588 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:44.175563 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:44.433500 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:44.433411 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:44.673907 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:44.673869 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" event={"ID":"2ec76a17-5431-4756-a824-862c354595c8","Type":"ContainerStarted","Data":"9720872459b7ecdaea62f99eee5a354f8e2b8c4ba7303d20000cf0d87249aae6"} Oct 31 00:46:44.674181 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:44.674159 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" event={"ID":"2ec76a17-5431-4756-a824-862c354595c8","Type":"ContainerStarted","Data":"b7202743eaf85508f06d19ca86ae0e8b1111f342b6b9a7c08c59cbe3b46e23d7"} Oct 31 00:46:44.676898 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:44.676864 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" event={"ID":"b22ec283-637b-4dd0-add9-76777f64e77c","Type":"ContainerStarted","Data":"a0e48461a68564f3f5616b4e84451907a7dca742aaa5a86a9e9b91ad867ad0d8"} Oct 31 00:46:44.680328 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:44.680298 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" event={"ID":"a3eab170-3ff7-48a5-8f32-9acfcb6db28b","Type":"ContainerStarted","Data":"6f60b4c9e8622704ba2f3440059aadb872b0bab7cfc5020331323d5f4a0d81f6"} Oct 31 00:46:44.687775 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:44.687687 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" event={"ID":"6a11f261-4cc7-4ca9-8b5f-8920883108b4","Type":"ContainerStarted","Data":"0a7dec6811c1d96b62f3a9d7b834bef41758df021a32c6f1e4dc9059099272f9"} Oct 31 00:46:44.688045 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:44.688020 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" event={"ID":"6a11f261-4cc7-4ca9-8b5f-8920883108b4","Type":"ContainerStarted","Data":"046eba82f2e738d8e1b6e122515eb314b0305f5d2f384654466f146218eab092"} Oct 31 00:46:44.756689 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:44.748321 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" podStartSLOduration=2.748297302 podStartE2EDuration="2.748297302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:44.708629665 +0000 UTC m=+800.737810056" watchObservedRunningTime="2026-10-31 00:46:44.748297302 +0000 UTC m=+800.777477693" Oct 31 00:46:44.799063 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:44.798993 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" podStartSLOduration=2.7989706610000002 podStartE2EDuration="2.798970661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:44.752286598 +0000 UTC m=+800.781466988" watchObservedRunningTime="2026-10-31 00:46:44.798970661 +0000 UTC m=+800.828151052" Oct 31 00:46:44.832855 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:44.832786 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" podStartSLOduration=1.832768389 podStartE2EDuration="1.832768389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:44.83223182 +0000 UTC m=+800.861412211" watchObservedRunningTime="2026-10-31 00:46:44.832768389 +0000 UTC m=+800.861948780" Oct 31 00:46:44.845658 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:44.832981 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:44.846525 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:44.846464 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" podStartSLOduration=2.846440643 podStartE2EDuration="2.846440643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:44.789235516 +0000 UTC m=+800.818415907" watchObservedRunningTime="2026-10-31 00:46:44.846440643 +0000 UTC m=+800.875621035" Oct 31 00:46:45.044659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:45.024504 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:45.192894 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:45.157934 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:45.192894 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:45.159313 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:46:45.192894 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:45.159380 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:46:45.314836 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:45.314736 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:45.315195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:45.315174 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:46:45.315368 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:45.315344 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:46:45.357662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:45.351525 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:45.387405 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:45.381607 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:45.387405 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:45.381767 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:46:45.387405 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:45.381820 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" podUID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:46:45.828604 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:45.828567 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:46:45.829320 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:45.829285 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" podUID="2ec76a17-5431-4756-a824-862c354595c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:46:46.050207 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:46.050155 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:46.050848 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:46.050818 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 31 00:46:46.064172 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:46.064133 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:46.065277 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:46.065247 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:46.069734 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:46.069698 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:46.112658 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:46.112546 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:46:46.112989 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:46.112962 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:46:46.218672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:46.213074 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:46.304285 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:46.304254 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:46:46.304590 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:46.304552 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:46:46.363274 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:46.363179 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:46:46.363570 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:46.363518 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" podUID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:46:46.624721 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:46.618727 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 00:46:46.832661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:46.829797 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:46:46.832661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:46.829861 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" podUID="2ec76a17-5431-4756-a824-862c354595c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:46:46.898606 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:46.898452 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:47.074963 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:47.072745 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:47.074963 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:47.073153 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:46:47.110543 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:47.110509 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:46:47.110876 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:47.110845 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:46:47.182747 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:47.179168 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:47.356877 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:47.356821 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:47.414934 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:47.414901 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:46:47.415218 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:47.415182 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:46:47.415826 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:47.415804 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:46:47.416003 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:47.415983 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" podUID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:46:47.611661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:47.608809 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:47.620644 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:47.620582 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:47.623081 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:47.623048 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:47.904276 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:47.904175 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:46:47.911756 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:47.906316 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:46:47.948662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:47.948609 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:46:47.948966 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:47.948931 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" podUID="2ec76a17-5431-4756-a824-862c354595c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:46:48.008327 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:46:48.000397 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:46:48.038674 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.038609 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:48.113216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.113171 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 00:46:48.115096 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.115061 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 00:46:48.118018 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.117984 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:46:48.140798 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.120711 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:46:48.144994 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.144952 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:46:48.145228 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.141097 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:48.163985 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.163904 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:48.483348 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.451771 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:46:48.483348 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.451848 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:46:48.485774 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.485738 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:46:48.486004 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.485979 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" podUID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:46:48.627661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.615196 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:48.627661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.616153 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:46:48.638357 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.638272 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 00:46:48.658128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.658098 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:48.825719 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.825042 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:46:48.825719 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.825103 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" podUID="2ec76a17-5431-4756-a824-862c354595c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:46:48.924660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.910768 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:48.987660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:48.987255 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:49.108711 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.087845 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:49.108711 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.088341 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:49.108711 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.088570 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:49.120663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.118114 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:46:49.120663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.118175 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:46:49.120663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.120054 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:49.120663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.120432 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:46:49.161660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.160916 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:49.306937 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.306901 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:46:49.311949 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.311908 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:46:49.343197 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.343166 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:46:49.390356 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.390262 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:46:49.390691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.390657 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" podUID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:46:49.642154 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.642055 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:49.834230 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.834192 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:46:49.834501 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.834469 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" podUID="2ec76a17-5431-4756-a824-862c354595c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:46:49.852851 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.852818 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:49.911821 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:49.911722 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:50.005661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.004748 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:46:50.071668 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.070746 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:50.100666 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.090789 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:50.108657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.103776 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:46:50.108657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.104160 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 00:46:50.108657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.104486 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:50.152689 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.128836 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:46:50.152689 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.128888 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:46:50.169818 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.168793 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:50.274390 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.273736 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:50.326025 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.325990 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:46:50.326323 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.326293 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:46:50.365520 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.365067 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:46:50.365520 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.365142 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" podUID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:46:50.434325 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.434227 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:50.650813 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.650759 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:50.786663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.774874 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:50.838671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.832089 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:46:50.839018 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.838977 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" podUID="2ec76a17-5431-4756-a824-862c354595c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:46:50.962662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.959549 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:50.963374 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:50.963339 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:46:51.113796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:51.113145 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:46:51.113796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:51.113204 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:46:51.455722 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:51.435046 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:46:51.455722 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:51.435110 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" podUID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:46:51.494672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:51.492787 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:46:51.494672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:51.492849 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:46:51.650003 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:51.649959 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 31 00:46:51.664305 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:51.662226 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:51.681984 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:51.681940 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:46:51.798431 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:51.798393 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:51.825144 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:51.825107 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:46:51.825420 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:51.825385 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" podUID="2ec76a17-5431-4756-a824-862c354595c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:46:51.985658 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:51.981058 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:52.081717 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.081290 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg"] Oct 31 00:46:52.117720 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.109010 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:46:52.133661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.118289 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:46:52.134855 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.134811 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:52.145667 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.135125 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:46:52.153284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.135620 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:52.153284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.135694 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:52.223687 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.200753 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:52.307216 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.307160 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:52.377427 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.360364 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:46:52.403971 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.377741 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:46:52.532661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.361319 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.532661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.361205 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg"] Oct 31 00:46:52.532661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.360966 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:46:52.532661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.450454 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:52.532661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.513901 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" podUID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:46:52.534351 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.534325 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1d4a7e\"/\"kube-root-ca.crt\"" Oct 31 00:46:52.535556 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.534551 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1d4a7e\"/\"default-dockercfg-f794t\"" Oct 31 00:46:52.535875 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.534708 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1d4a7e\"/\"openshift-service-ca.crt\"" Oct 31 00:46:52.536008 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.534774 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1d4a7e\"/\"distinct-2-w3-1d4a7e-kserve-self-signed-certs\"" Oct 31 00:46:52.608575 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.608522 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2h9w\" (UniqueName: \"kubernetes.io/projected/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-kube-api-access-z2h9w\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.608934 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.608914 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-model-cache\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.613660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.609172 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-tmp-dir\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.614008 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.613970 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-home\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.614326 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.614287 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-tls-certs\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.614530 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.614514 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-dshm\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.796799 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.788995 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-home\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.796799 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.789133 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-tls-certs\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.796799 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.789184 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-dshm\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.796799 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.789266 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2h9w\" (UniqueName: \"kubernetes.io/projected/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-kube-api-access-z2h9w\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.796799 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.789347 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-model-cache\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.796799 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.789470 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-tmp-dir\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.796799 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.790025 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-tmp-dir\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.797311 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.797073 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:52.805660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.797946 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:46:52.846078 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.846043 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-home\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.861100 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.861062 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-model-cache\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.892661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.881000 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-dshm\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.892921 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.888705 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-tls-certs\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:52.893018 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.889095 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:46:52.893168 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.893131 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" podUID="2ec76a17-5431-4756-a824-862c354595c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:46:53.000673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:52.998421 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:46:53.186723 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.157770 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 00:46:53.186723 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.158272 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:46:53.186723 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.171125 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:46:53.186723 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.171196 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:46:53.219115 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.188843 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:53.219115 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.189331 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:46:53.223664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.219457 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:53.223664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.221041 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:53.223664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.223481 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2h9w\" (UniqueName: \"kubernetes.io/projected/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-kube-api-access-z2h9w\") pod \"distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:53.241118 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.241089 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:53.241499 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.241470 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:46:53.267917 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.266361 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:53.323657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.316159 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:53.323657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.316603 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:53.370661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.345171 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:46:53.372678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.370909 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:46:53.372678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.370849 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:46:53.372678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.371315 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" podUID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:46:53.610331 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.599567 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 00:46:53.683822 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.665060 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:53.683822 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.665516 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:46:53.782107 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.771720 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:53.886684 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.847145 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:53.894662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.860471 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:46:53.966767 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.966713 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" podUID="2ec76a17-5431-4756-a824-862c354595c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:46:53.998566 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.897524 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:53.998922 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:53.998877 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:54.235732 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.194890 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:46:54.235732 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.208742 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:54.251455 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.248656 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:54.251455 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.249152 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 31 00:46:54.305344 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.305017 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:46:54.305344 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.305092 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:46:54.529049 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.410779 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:46:54.529049 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.410850 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" podUID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:46:54.529049 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.410933 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:46:54.529049 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.410956 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:46:54.603780 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.603723 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:54.669436 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.669388 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:54.708078 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.707993 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:54.826902 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.826813 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:46:54.827178 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.827152 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" podUID="2ec76a17-5431-4756-a824-862c354595c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:46:54.856574 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.841750 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:54.890166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.882932 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:46:54.927673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:54.913576 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:55.059662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.027441 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:55.539260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.111862 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:46:55.539260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.472764 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:46:55.539260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.228903 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 00:46:55.539260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.200736 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:55.539260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.322974 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:46:55.539260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.486926 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:46:55.539260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.301408 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:55.539260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.228936 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 00:46:55.539260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.258992 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 31 00:46:55.539260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.259026 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:46:55.539260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.367794 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:46:55.539260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.509228 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" podUID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:46:55.539260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.416179 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:55.539260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.523978 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg"] Oct 31 00:46:55.659873 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.637715 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 00:46:55.659873 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.654419 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 00:46:55.663949 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.663920 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:46:55.696657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.691694 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:46:55.829941 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.829906 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:46:56.013889 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.013781 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" podUID="2ec76a17-5431-4756-a824-862c354595c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:46:56.014709 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:55.984443 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:56.125122 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.125089 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:46:56.154661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.125314 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:46:56.431586 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.425364 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:56.737096 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.485812 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 00:46:56.737096 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.490761 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:56.739700 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.490813 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 00:46:56.739992 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.521082 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:56.740623 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.724415 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:56.741221 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.592797 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:56.741326 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.741286 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Oct 31 00:46:56.746481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.729539 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:56.746481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.742824 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:56.746481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.742900 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:46:56.746481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.742920 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" event={"ID":"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc","Type":"ContainerStarted","Data":"189197a0b832c37133b573880f57ab2662000b6ee2943b0cc827dad25ff4a18a"} Oct 31 00:46:56.746481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.742985 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:56.769654 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.754290 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:56.769654 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.757107 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:56.794084 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.794051 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:46:56.804108 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.804077 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:46:56.814621 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.814594 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:46:56.874411 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.873627 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:46:56.934799 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.934763 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:46:56.951809 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.951779 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:46:56.975666 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.971924 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:46:56.991069 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:56.990992 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:46:57.066663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:57.063751 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:57.117667 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:57.115923 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:46:57.117667 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:57.115998 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:46:57.171854 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:57.171826 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:46:57.229704 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:57.229184 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:57.327843 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:57.327745 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:57.328147 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:57.328118 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:57.527664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:57.523921 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" event={"ID":"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc","Type":"ContainerStarted","Data":"75442217dff62ea366113494e6eb83bf9b5063062802f679b55cee31eeb1cd67"} Oct 31 00:46:57.562672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:57.558612 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podStartSLOduration=5.558593652 podStartE2EDuration="5.558593652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:57.556044915 +0000 UTC m=+813.585225308" watchObservedRunningTime="2026-10-31 00:46:57.558593652 +0000 UTC m=+813.587774045" Oct 31 00:46:57.806662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:57.793942 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:57.815681 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:57.812267 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:57.843824 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:57.843721 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:46:57.967121 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:57.967088 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:57.998935 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:57.974863 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx"] Oct 31 00:46:58.021671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.020857 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx"] Oct 31 00:46:58.021671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.020990 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.104809 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.075049 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-44bc67\"/\"distinct-2-w5-44bc67-kserve-self-signed-certs\"" Oct 31 00:46:58.155887 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.117947 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-dshm\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.155887 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.118040 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-home\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.155887 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.119089 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:58.341394 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.296371 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9npw\" (UniqueName: \"kubernetes.io/projected/2f863a10-a9a8-4894-ac86-2f057371b472-kube-api-access-t9npw\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.341394 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.296459 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-dshm\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.341394 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.296540 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-home\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.341394 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.296580 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f863a10-a9a8-4894-ac86-2f057371b472-tls-certs\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.341394 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.296658 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-tmp-dir\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.341394 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.296783 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-model-cache\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.341394 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.301006 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-home\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.341394 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.329145 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:58.341394 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.329203 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:58.380743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.374659 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:58.380743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.374727 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:46:58.380743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.374746 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:58.404662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.401054 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-model-cache\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.404662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.401143 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9npw\" (UniqueName: \"kubernetes.io/projected/2f863a10-a9a8-4894-ac86-2f057371b472-kube-api-access-t9npw\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.404662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.401324 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f863a10-a9a8-4894-ac86-2f057371b472-tls-certs\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.404662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.401382 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-tmp-dir\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.404662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.404485 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-model-cache\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.407559 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.405516 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-tmp-dir\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.452658 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.452246 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f863a10-a9a8-4894-ac86-2f057371b472-tls-certs\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.462338 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.462301 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9npw\" (UniqueName: \"kubernetes.io/projected/2f863a10-a9a8-4894-ac86-2f057371b472-kube-api-access-t9npw\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.517115 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.517038 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-dshm\") pod \"distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.643938 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.643855 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:46:58.659546 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.659516 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:58.671754 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.671727 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:46:58.770270 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.769004 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:58.790974 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.790888 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:46:58.960491 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:58.960406 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:59.002790 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:59.002755 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx"] Oct 31 00:46:59.003699 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:46:59.003663 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f863a10_a9a8_4894_ac86_2f057371b472.slice/crio-87b8111251cc88a0f023db124c84a0addd5a63e1d1ab542e8445bcba8442af36 WatchSource:0}: Error finding container 87b8111251cc88a0f023db124c84a0addd5a63e1d1ab542e8445bcba8442af36: Status 404 returned error can't find the container with id 87b8111251cc88a0f023db124c84a0addd5a63e1d1ab542e8445bcba8442af36 Oct 31 00:46:59.638623 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:59.638590 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:59.644704 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:59.644673 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" event={"ID":"2f863a10-a9a8-4894-ac86-2f057371b472","Type":"ContainerStarted","Data":"744b19da8f245e6fb4da7b00b6fa9b1b74d6652c41f462af46d954a1999b5df1"} Oct 31 00:46:59.644890 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:59.644713 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" event={"ID":"2f863a10-a9a8-4894-ac86-2f057371b472","Type":"ContainerStarted","Data":"87b8111251cc88a0f023db124c84a0addd5a63e1d1ab542e8445bcba8442af36"} Oct 31 00:46:59.663561 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:46:59.663507 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podStartSLOduration=2.663494462 podStartE2EDuration="2.663494462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:59.661166501 +0000 UTC m=+815.690346918" watchObservedRunningTime="2026-10-31 00:46:59.663494462 +0000 UTC m=+815.692674852" Oct 31 00:47:00.417090 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:00.417058 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:00.644649 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:00.644614 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:47:01.132059 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:01.132022 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:01.726401 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:01.726369 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:02.011375 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:02.011343 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:02.338098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:02.338018 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:02.745164 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:02.745113 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:03.150615 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:03.150535 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:03.289242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:03.289201 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:47:03.320387 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:03.320357 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:47:03.677595 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:03.677568 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:47:03.951240 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:03.951154 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:04.618409 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:04.618373 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:05.231978 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:05.231943 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:05.940587 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:05.940553 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:06.543428 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:06.543395 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:07.145022 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:07.144980 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:07.748442 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:07.748410 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:08.531269 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:08.531238 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:08.650965 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:08.650934 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:47:08.708685 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:08.708353 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:47:08.740770 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:08.740625 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:47:09.391823 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:09.391784 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:11.304767 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:11.304713 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:11.960819 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:11.960763 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:11.979791 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:11.979742 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:12.207299 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:12.207253 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:12.253165 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:12.253114 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:47:12.518871 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:12.518757 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:12.664661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:12.643816 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:12.754660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:12.752762 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 00:47:12.754660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:12.753969 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:12.765709 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:12.758713 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:47:12.821051 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:12.820950 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:47:13.014943 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.014897 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" podUID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:13.015201 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:12.899033 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:47:13.016005 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:12.833734 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:13.029664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.026098 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:47:13.087793 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.077693 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 00:47:13.108131 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.098199 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng"] Oct 31 00:47:13.157912 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.157880 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:47:13.158133 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.158108 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.165662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.158206 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:47:13.205311 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.203906 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-workload-certs\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.205311 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.203946 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-workload-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.205311 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.204009 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-data\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.205311 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.204040 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-podinfo\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.205311 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.204105 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-credential-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.205311 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.204149 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-envoy\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.205311 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.204259 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-token\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.205311 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.204288 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istiod-ca-cert\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.205311 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.204371 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7gvv\" (UniqueName: \"kubernetes.io/projected/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-kube-api-access-c7gvv\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.210696 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.208165 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm"] Oct 31 00:47:13.213227 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.213137 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:47:13.298150 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.297579 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:13.321671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.308873 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-credential-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.321671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.308941 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-envoy\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.321671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.309032 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-token\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.321671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.309060 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istiod-ca-cert\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.321671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.309142 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7gvv\" (UniqueName: \"kubernetes.io/projected/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-kube-api-access-c7gvv\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.321671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.309273 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-workload-certs\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.321671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.309298 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-workload-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.321671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.309345 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-data\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.321671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.309375 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-podinfo\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.324602 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.323213 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istiod-ca-cert\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.324602 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.324273 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-credential-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.334020 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.333985 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-workload-certs\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.334312 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.334283 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-workload-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.341923 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.341860 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:47:13.346721 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.346696 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng"] Oct 31 00:47:13.346950 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.346936 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:47:13.366298 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.366268 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.407658 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.383658 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm"] Oct 31 00:47:13.425325 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.389914 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-data\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.453258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.412160 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7gvv\" (UniqueName: \"kubernetes.io/projected/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-kube-api-access-c7gvv\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.482660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.432310 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-envoy\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.482660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.433171 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-podinfo\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.482660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.450684 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4"] Oct 31 00:47:13.492436 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.453181 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-token\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.518416 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.518040 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.538660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.519530 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:13.540624 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.540594 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:47:13.567328 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.557215 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w"] Oct 31 00:47:13.647995 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.646029 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:13.651522 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.648878 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:13.651522 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.649577 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-workload-certs\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.651522 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.649617 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-istio-data\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.732300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.688778 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 00:47:13.732300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.698875 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:47:13.732300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.698930 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:47:13.732300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.699032 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:13.732300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.710799 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-credential-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.732300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.710898 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-workload-certs\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.732300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.710973 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a619a89-f559-4918-bbbb-2cf358157e77-istio-token\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.732300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.711015 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-istio-envoy\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.732300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.711046 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06a6761a-4e9e-4675-8dec-dea23f865635-istio-podinfo\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.732300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.729012 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:13.770145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.743964 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-credential-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.770145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.744039 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5hz9\" (UniqueName: \"kubernetes.io/projected/06a6761a-4e9e-4675-8dec-dea23f865635-kube-api-access-b5hz9\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.770145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.744131 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-workload-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.770145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.744211 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-workload-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.770145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.744311 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-istio-data\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.770145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.744357 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mstdx\" (UniqueName: \"kubernetes.io/projected/8a619a89-f559-4918-bbbb-2cf358157e77-kube-api-access-mstdx\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.770145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.744432 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-istio-envoy\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.770145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.744511 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a619a89-f559-4918-bbbb-2cf358157e77-istiod-ca-cert\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.770145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.744539 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06a6761a-4e9e-4675-8dec-dea23f865635-istiod-ca-cert\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.770145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.747783 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a619a89-f559-4918-bbbb-2cf358157e77-istio-podinfo\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.770145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.747870 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06a6761a-4e9e-4675-8dec-dea23f865635-istio-token\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.770145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.754770 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 00:47:13.770145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.765030 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 00:47:13.770145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.766870 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:47:13.770145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.766935 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:47:13.770145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.767025 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:13.772876 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.772856 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.849527 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-workload-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.849592 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c9v6\" (UniqueName: \"kubernetes.io/projected/55e0cdd8-4547-44da-b720-b2e35ee69846-kube-api-access-7c9v6\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.849660 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-workload-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.849703 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-workload-certs\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.849738 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-envoy\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.849766 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-token\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.849816 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-istio-data\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.849842 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mstdx\" (UniqueName: \"kubernetes.io/projected/8a619a89-f559-4918-bbbb-2cf358157e77-kube-api-access-mstdx\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.849887 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-istio-envoy\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.849922 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a619a89-f559-4918-bbbb-2cf358157e77-istiod-ca-cert\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.849945 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06a6761a-4e9e-4675-8dec-dea23f865635-istiod-ca-cert\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.849991 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a619a89-f559-4918-bbbb-2cf358157e77-istio-podinfo\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.850039 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06a6761a-4e9e-4675-8dec-dea23f865635-istio-token\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.850092 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-workload-certs\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.850116 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-istio-data\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.850178 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-credential-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.850223 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-workload-certs\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.850251 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-workload-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.850288 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e0cdd8-4547-44da-b720-b2e35ee69846-istiod-ca-cert\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.850339 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a619a89-f559-4918-bbbb-2cf358157e77-istio-token\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.850365 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-istio-envoy\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.850398 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06a6761a-4e9e-4675-8dec-dea23f865635-istio-podinfo\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.850422 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-data\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.850537 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-credential-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.850584 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5hz9\" (UniqueName: \"kubernetes.io/projected/06a6761a-4e9e-4675-8dec-dea23f865635-kube-api-access-b5hz9\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.850624 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-podinfo\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:13.857490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.850685 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-credential-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:13.879910 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.876471 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-workload-certs\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.879910 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.876876 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-workload-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.879910 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.877155 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-workload-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.879910 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.877522 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-istio-data\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.879910 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.878007 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-istio-data\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.879910 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.878550 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-credential-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:13.879910 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.878827 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-workload-certs\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:13.879910 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.879874 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-credential-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:14.033054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.921404 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a619a89-f559-4918-bbbb-2cf358157e77-istiod-ca-cert\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:14.033054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.938042 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:47:14.033054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.938098 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" podUID="2ec76a17-5431-4756-a824-862c354595c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:14.033054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.952120 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-istio-envoy\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:14.033054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.960925 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-workload-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.033054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.960974 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e0cdd8-4547-44da-b720-b2e35ee69846-istiod-ca-cert\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.033054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.961024 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-data\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.033054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.961162 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-podinfo\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.033054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.961196 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-credential-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.033054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.961245 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c9v6\" (UniqueName: \"kubernetes.io/projected/55e0cdd8-4547-44da-b720-b2e35ee69846-kube-api-access-7c9v6\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.033054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.961326 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-workload-certs\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.033054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.961377 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-envoy\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.033054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.961399 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-token\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.033054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:13.989357 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a619a89-f559-4918-bbbb-2cf358157e77-istio-token\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:14.066242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.037682 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06a6761a-4e9e-4675-8dec-dea23f865635-istiod-ca-cert\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:14.066242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.038210 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-data\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.066242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.045982 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-workload-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.066242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.053792 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:14.066242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.065406 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-workload-certs\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.091763 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.089358 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e0cdd8-4547-44da-b720-b2e35ee69846-istiod-ca-cert\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.101665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.093557 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-credential-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.107239 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.105373 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:14.107239 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.106418 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4"] Oct 31 00:47:14.107239 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.116481 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w"] Oct 31 00:47:14.107239 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.121262 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:14.213862 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.213806 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:14.258025 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.257995 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-envoy\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.276654 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.273473 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5hz9\" (UniqueName: \"kubernetes.io/projected/06a6761a-4e9e-4675-8dec-dea23f865635-kube-api-access-b5hz9\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:14.276654 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.274236 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mstdx\" (UniqueName: \"kubernetes.io/projected/8a619a89-f559-4918-bbbb-2cf358157e77-kube-api-access-mstdx\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:14.301195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.298749 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded" Oct 31 00:47:14.340927 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.302216 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:14.340927 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.334920 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w"] Oct 31 00:47:14.345674 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.344243 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.504373 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.504332 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w"] Oct 31 00:47:14.536728 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.526756 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jnhs\" (UniqueName: \"kubernetes.io/projected/e5fc25c3-483d-4644-b774-815f60af2025-kube-api-access-2jnhs\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.547961 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.547931 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-istio-data\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.548553 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.548534 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-istio-envoy\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.548911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.548893 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-credential-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.549075 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.549061 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5fc25c3-483d-4644-b774-815f60af2025-istiod-ca-cert\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.549316 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.549298 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5fc25c3-483d-4644-b774-815f60af2025-istio-token\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.549500 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.549486 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5fc25c3-483d-4644-b774-815f60af2025-istio-podinfo\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.549841 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.549819 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-workload-certs\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.549984 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.549970 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-workload-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.550945 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.543844 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 31 00:47:14.551673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.544836 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-istio-envoy\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:14.555044 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.555023 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06a6761a-4e9e-4675-8dec-dea23f865635-istio-podinfo\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:14.558187 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.558166 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-podinfo\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.576777 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.576738 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c9v6\" (UniqueName: \"kubernetes.io/projected/55e0cdd8-4547-44da-b720-b2e35ee69846-kube-api-access-7c9v6\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.589871 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.586184 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06a6761a-4e9e-4675-8dec-dea23f865635-istio-token\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:14.613108 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.613059 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 00:47:14.631107 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.631068 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-token\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.635585 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.635559 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:14.649665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.641342 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a619a89-f559-4918-bbbb-2cf358157e77-istio-podinfo\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:14.657388 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.657338 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:14.739659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.736224 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:14.752688 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.736994 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-credential-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.752688 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.745525 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5fc25c3-483d-4644-b774-815f60af2025-istiod-ca-cert\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.752688 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.745561 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5fc25c3-483d-4644-b774-815f60af2025-istio-token\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.752688 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.745659 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5fc25c3-483d-4644-b774-815f60af2025-istio-podinfo\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.752688 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.745742 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-workload-certs\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.752688 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.745791 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-workload-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.752688 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.745853 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnhs\" (UniqueName: \"kubernetes.io/projected/e5fc25c3-483d-4644-b774-815f60af2025-kube-api-access-2jnhs\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.752688 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.746022 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-istio-data\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.752688 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.746121 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-istio-envoy\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.760019 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.758569 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:47:14.761122 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.761101 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:47:14.774189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.762963 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 00:47:14.774189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.765722 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:14.774189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.767109 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:47:14.774189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.772211 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-workload-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.774189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.772496 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-workload-certs\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.774189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.737502 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-credential-socket\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.774189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.772999 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-istio-data\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.774189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.773190 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-istio-envoy\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.775183 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.775126 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5fc25c3-483d-4644-b774-815f60af2025-istiod-ca-cert\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:14.775568 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.775475 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:47:14.775568 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.775515 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:14.839491 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.838172 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:14.874691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:14.854830 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:47:15.137822 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:15.137714 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:15.167715 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:15.167675 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5fc25c3-483d-4644-b774-815f60af2025-istio-token\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:15.168915 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:15.168895 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:47:15.176163 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:15.176141 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:47:15.191515 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:15.187857 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jnhs\" (UniqueName: \"kubernetes.io/projected/e5fc25c3-483d-4644-b774-815f60af2025-kube-api-access-2jnhs\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:15.227672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:15.225362 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5fc25c3-483d-4644-b774-815f60af2025-istio-podinfo\") pod \"e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:15.415666 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:15.406151 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:15.463812 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:15.417254 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:15.466327 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:15.466298 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:15.476286 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:15.476046 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:15.838749 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:15.829260 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 00:47:15.946150 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:15.946108 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:47:15.956126 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:15.956081 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng"] Oct 31 00:47:16.007941 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.007863 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:47:16.008491 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:47:16.008457 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69ce0fa_acb7_4e5a_b578_6333d4cf0ae4.slice/crio-f94f6948c3a43a81b27cac0bd1f3578c2252cd22986b0a0c616a0ce477ba82e4 WatchSource:0}: Error finding container f94f6948c3a43a81b27cac0bd1f3578c2252cd22986b0a0c616a0ce477ba82e4: Status 404 returned error can't find the container with id f94f6948c3a43a81b27cac0bd1f3578c2252cd22986b0a0c616a0ce477ba82e4 Oct 31 00:47:16.009471 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.009451 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:47:16.050321 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.044075 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:16.050321 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.044227 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:16.050321 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.044316 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:16.157508 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.157426 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:16.159735 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.159656 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm"] Oct 31 00:47:16.177152 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.166424 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:16.177152 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.166511 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:16.177152 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.166553 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:16.362411 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.358746 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" event={"ID":"8a619a89-f559-4918-bbbb-2cf358157e77","Type":"ContainerStarted","Data":"e8719fd195b7e49cd9de8bee1a91870de5ec13876861944e20ae4dc30513c641"} Oct 31 00:47:16.362411 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.361494 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" event={"ID":"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4","Type":"ContainerStarted","Data":"f94f6948c3a43a81b27cac0bd1f3578c2252cd22986b0a0c616a0ce477ba82e4"} Oct 31 00:47:16.371528 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.371500 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4"] Oct 31 00:47:16.431713 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.431672 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w"] Oct 31 00:47:16.435006 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.434958 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:16.435127 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.435097 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:16.435212 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.435193 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:16.445054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.442773 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:16.445054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.442923 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:16.445054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.443015 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:16.453884 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.453433 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w"] Oct 31 00:47:16.465706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.465091 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:16.465706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.465293 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:16.465706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.465396 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:16.880714 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:16.880462 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:17.391110 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.389024 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" event={"ID":"e5fc25c3-483d-4644-b774-815f60af2025","Type":"ContainerStarted","Data":"6fa1b78056d929bb107371a158e3918f3ef5908eb02a375be0e1f3a5935d4daa"} Oct 31 00:47:17.391110 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.389078 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" event={"ID":"e5fc25c3-483d-4644-b774-815f60af2025","Type":"ContainerStarted","Data":"fe405f4ce8d2fca74caa42ea587ffb71ad8b9ae791dd3c62c1675d5512a72883"} Oct 31 00:47:17.394673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.391721 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" event={"ID":"06a6761a-4e9e-4675-8dec-dea23f865635","Type":"ContainerStarted","Data":"87d40eed09e0c1f606cb69ac3819cefb70bb9d9e4d5c6d3abbfe3267182ef87c"} Oct 31 00:47:17.394673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.391769 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" event={"ID":"06a6761a-4e9e-4675-8dec-dea23f865635","Type":"ContainerStarted","Data":"93fa8053d344c13f516387883f62949ec6ce9589617800b83f165f5dea769ee9"} Oct 31 00:47:17.394673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.394292 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" event={"ID":"8a619a89-f559-4918-bbbb-2cf358157e77","Type":"ContainerStarted","Data":"fce4eb71bd9706f20425a86690448aa91797027e9680f4f323fb185d698fc2e4"} Oct 31 00:47:17.397143 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.397117 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" event={"ID":"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4","Type":"ContainerStarted","Data":"bce6075ae55a7ca1f200f28d937d6f827e2581897165e8cfdfe357b770986ef2"} Oct 31 00:47:17.399214 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.399191 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" event={"ID":"55e0cdd8-4547-44da-b720-b2e35ee69846","Type":"ContainerStarted","Data":"562e889c23398bd92b8c294c6094af7e8ba054dd567fb57b0e7f6c22164de97e"} Oct 31 00:47:17.399327 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.399229 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" event={"ID":"55e0cdd8-4547-44da-b720-b2e35ee69846","Type":"ContainerStarted","Data":"7cf1674a31414bce3397435d220879ac6db2bd982f86a9725dc3221985fd97ae"} Oct 31 00:47:17.415185 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.415118 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" podStartSLOduration=4.415102295 podStartE2EDuration="4.415102295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:17.411729618 +0000 UTC m=+833.440910008" watchObservedRunningTime="2026-10-31 00:47:17.415102295 +0000 UTC m=+833.444282685" Oct 31 00:47:17.438358 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.438292 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" podStartSLOduration=4.438270684 podStartE2EDuration="4.438270684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:17.433941887 +0000 UTC m=+833.463122281" watchObservedRunningTime="2026-10-31 00:47:17.438270684 +0000 UTC m=+833.467451077" Oct 31 00:47:17.462886 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.462596 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" podStartSLOduration=5.462574084 podStartE2EDuration="5.462574084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:17.461946779 +0000 UTC m=+833.491127168" watchObservedRunningTime="2026-10-31 00:47:17.462574084 +0000 UTC m=+833.491754475" Oct 31 00:47:17.477863 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.477831 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:17.481573 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.481545 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:47:17.481838 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.481812 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:47:17.519557 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.519490 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" podStartSLOduration=5.519468374 podStartE2EDuration="5.519468374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:17.492212977 +0000 UTC m=+833.521393368" watchObservedRunningTime="2026-10-31 00:47:17.519468374 +0000 UTC m=+833.548648765" Oct 31 00:47:17.613093 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:47:17.613050 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:47:17.656719 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.652616 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:17.657549 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.657518 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:47:17.657792 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.657764 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:47:17.659193 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.659168 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:17.678377 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.678346 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:17.678734 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.678713 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:47:17.678896 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.678872 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 00:47:17.744138 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.741364 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:17.744138 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.741538 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:47:17.744138 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.741587 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:47:17.857411 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.857376 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:17.857790 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.857768 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:47:17.857951 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:17.857929 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 00:47:18.343213 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:18.343179 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:18.487373 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:18.487336 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:47:18.487736 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:18.487699 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:47:18.656870 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:18.656783 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:47:18.657141 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:18.657109 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:47:18.687079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:18.687041 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:47:18.687352 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:18.687322 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 00:47:18.745823 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:18.741101 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:47:18.745823 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:18.741165 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:47:18.776672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:18.773399 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:18.776672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:18.773749 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 00:47:18.804242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:18.804125 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:18.861672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:18.859779 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:18.861672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:18.861255 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:47:18.861672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:18.861296 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 00:47:18.900911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:18.900828 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:18.960842 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:18.960720 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:19.120850 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.120801 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:19.300603 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.300562 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:19.425927 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.425881 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:19.501111 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.501073 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:47:19.515667 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.515348 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:47:19.654292 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.654188 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:19.656795 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.656761 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:47:19.657074 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.657048 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:47:19.676275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.676239 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:47:19.676542 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.676514 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 00:47:19.681760 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.681718 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:19.754655 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.753826 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:47:19.754655 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.753883 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:47:19.755884 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.755788 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:19.778564 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.776787 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:19.778899 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.778855 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:19.779336 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.778855 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:19.780054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.779526 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:47:19.780249 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.780232 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:47:19.814218 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.814170 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:19.814930 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.814899 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:47:19.834146 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.834081 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" podStartSLOduration=6.834059419 podStartE2EDuration="6.834059419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:17.520344195 +0000 UTC m=+833.549524577" watchObservedRunningTime="2026-10-31 00:47:19.834059419 +0000 UTC m=+835.863239818" Oct 31 00:47:19.895661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.867034 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:47:19.895661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.867098 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 00:47:19.950735 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.922765 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:19.950735 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:19.923234 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:47:20.174659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.173781 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:20.174878 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.174826 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:47:20.229198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.229091 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:20.301798 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.301742 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:20.447413 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.447358 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:20.451807 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.451761 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:47:20.506690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.498850 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:47:20.506690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.498925 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:47:20.506690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.502424 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:47:20.506690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.502488 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:47:20.589026 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.588986 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:20.722716 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.692466 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:20.722716 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.696610 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:47:20.725668 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.724376 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:47:20.725668 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.724456 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:47:20.731916 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.731782 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:47:20.731916 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.731865 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 00:47:20.758862 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.758764 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:47:20.759128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.759097 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:47:20.783706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.782773 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:20.784032 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.783987 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 00:47:20.785789 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.785755 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:47:20.810549 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.810510 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:20.833859 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.833796 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:20.863774 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.861295 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:47:20.890946 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.889097 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:20.902450 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.897679 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:47:20.902450 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.897751 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 00:47:20.924843 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.924786 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:20.926006 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.925965 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:47:20.971651 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:20.969691 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:47:21.045238 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.045134 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:21.289695 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.289653 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:47:21.290672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.290647 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:47:21.308542 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.308433 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 31 00:47:21.326265 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.326065 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:21.457060 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.449010 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:21.468795 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.453953 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:21.492685 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.486469 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:47:21.492685 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.486525 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:47:21.594952 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.594852 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:21.649668 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.649606 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:47:21.649955 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.649924 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:47:21.691052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.691016 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:47:21.691514 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.691483 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 00:47:21.699348 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.699302 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:21.750761 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.750724 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:47:21.775648 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.775396 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:47:21.791162 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.791085 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:21.791773 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.791732 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 00:47:21.863805 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.863720 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:47:21.864231 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.864202 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 00:47:21.864410 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.863729 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:21.911684 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.904795 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:21.911684 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.905205 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:21.911684 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.905491 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:47:21.936581 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.936526 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:47:21.937200 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.937168 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:47:21.937703 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:21.937685 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:47:22.048655 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.039778 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:22.330657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.330537 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:22.342665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.332970 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:22.354704 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.349960 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:22.514829 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.486019 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:47:22.540664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.515469 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:47:22.541414 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.488110 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:22.650122 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.650023 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:47:22.650354 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.650325 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:47:22.679124 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.679083 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:47:22.679373 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.679342 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 00:47:22.700313 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.700274 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 00:47:22.749305 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.749272 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:47:22.749598 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.749566 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:47:22.793758 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.793713 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 00:47:22.795104 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.795070 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:22.855298 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.853399 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:47:22.862331 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.862299 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:47:22.862573 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.862547 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 00:47:22.871232 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.871196 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:22.939079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.938971 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:22.939680 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.939626 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:47:22.940135 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.940115 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:47:22.957695 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.942455 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:23.074465 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:22.943334 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:23.076066 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.041345 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:23.076939 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.076915 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:47:23.105522 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.105488 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:23.444190 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.444148 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 31 00:47:23.457572 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.457533 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:23.506665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.505142 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:47:23.506665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.505209 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:47:23.558659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.558060 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:47:23.616334 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.616290 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:23.652038 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.652002 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:47:23.652312 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.652284 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:47:23.674813 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.674774 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:47:23.675152 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.675078 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 00:47:23.689096 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.688983 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:47:23.701026 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.700932 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:23.737380 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.737346 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:47:23.737670 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.737613 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:47:23.756741 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.752885 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:47:23.798955 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.796164 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:23.841457 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.841424 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:23.863497 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.863060 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:47:23.863497 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.863120 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 00:47:23.947741 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.945803 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:23.947741 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:23.946193 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 00:47:24.189756 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.161598 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:24.189756 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.163856 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:24.189756 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.164120 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:24.316787 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.316730 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:24.456009 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.455909 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:24.533077 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.532373 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:47:24.533077 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.532446 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:47:24.748806 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.737133 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:47:24.759321 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.759282 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:24.771463 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.771427 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:47:24.771769 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.771736 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:47:24.775401 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.775370 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:47:24.775605 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.775579 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:47:24.778024 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.777973 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:47:24.778194 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.778170 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 00:47:24.780759 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.780733 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:24.901673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.867298 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:47:24.909675 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.909326 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 00:47:24.909675 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.909263 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:24.994897 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.994819 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:24.995687 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:24.995654 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:25.005710 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.000231 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:47:25.297728 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.273357 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 00:47:25.297728 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.277373 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:25.334236 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.334192 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:25.414830 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.414770 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:25.462663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.437291 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:25.462663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.461014 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:25.490658 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.490597 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:25.513770 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.485618 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:47:25.514139 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.514029 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:47:25.695064 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.694975 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:47:25.695285 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.695260 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:47:25.697957 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.697931 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:47:25.698159 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.698136 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 00:47:25.770665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.768718 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:25.770665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.769722 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:47:25.770665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.769764 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:47:25.795666 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.794535 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:25.795666 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.794679 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:25.863781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.863747 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:47:25.864025 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.863999 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 00:47:25.997663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:25.996008 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:26.006069 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.001589 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:26.012746 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.007823 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:26.045376 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.045341 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:47:26.292092 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.291998 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:26.313668 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.292584 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:26.347659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.345302 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:26.384661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.357589 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:26.404664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.393514 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:47:26.417661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.417604 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:26.418555 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.418529 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:47:26.472659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.471724 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:26.696347 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.696262 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:47:26.696595 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.696565 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:47:26.697551 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.697523 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Oct 31 00:47:26.697852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.697831 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:47:26.697997 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.697978 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 00:47:26.738377 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.738343 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:47:26.738669 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.738621 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:47:26.770709 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.769848 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:26.839352 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.839312 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 00:47:26.840770 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.840748 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:47:26.857425 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.857395 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:47:26.892698 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.870230 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:47:26.892698 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.890956 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:47:26.892698 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.891022 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 00:47:26.943681 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:26.940358 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:47:27.038680 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.011111 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:27.039128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.013863 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:47:27.285671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.284931 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 00:47:27.311803 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.311709 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:27.336507 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.336466 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:27.340657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.337134 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:27.363774 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.363718 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:47:27.384697 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.365871 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded" Oct 31 00:47:27.419947 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.418622 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:47:27.483708 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.483231 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:47:27.483708 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.483293 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:47:27.638379 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.638266 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:27.653909 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.653867 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:47:27.654194 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.654152 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:47:27.677854 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.677825 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:47:27.678121 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.678084 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 00:47:27.741571 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.741516 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:47:27.741912 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.741859 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:47:27.844663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.843941 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:27.857651 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.857502 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:47:27.857651 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.857562 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 00:47:27.940513 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:27.940407 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:28.048793 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.048745 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:28.262986 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.262948 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr"] Oct 31 00:47:28.499308 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.469750 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:47:28.499308 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.472873 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:47:28.499308 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.472929 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:47:28.499308 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.473256 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.499308 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.475846 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 00:47:28.499308 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.484062 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 00:47:28.499308 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.484360 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:28.499308 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.485939 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:47:28.499308 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.485978 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:47:28.499308 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.486067 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 00:47:28.499308 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.487315 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:47:28.522307 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.518276 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr"] Oct 31 00:47:28.585659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.562504 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:28.594466 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.592572 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d89558b-7909-4d86-9305-720c63771d05-istio-podinfo\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.594466 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.592658 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d89558b-7909-4d86-9305-720c63771d05-istiod-ca-cert\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.594466 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.592716 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc2dn\" (UniqueName: \"kubernetes.io/projected/6d89558b-7909-4d86-9305-720c63771d05-kube-api-access-xc2dn\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.594466 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.592815 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-workload-certs\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.594466 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.592853 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-credential-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.594466 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.592885 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-istio-envoy\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.594466 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.593034 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-workload-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.594466 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.593109 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-istio-data\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.594466 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.593134 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d89558b-7909-4d86-9305-720c63771d05-istio-token\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.679662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.654411 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:28.780710 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.681426 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:47:28.842297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.835431 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 00:47:28.842297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.698173 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-workload-certs\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.842297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.697739 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-workload-certs\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.842297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.835591 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-credential-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.842297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.835628 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-istio-envoy\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.842297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.835821 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-workload-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.842297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.835902 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-istio-data\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.842297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.835942 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d89558b-7909-4d86-9305-720c63771d05-istio-token\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.842297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.836067 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d89558b-7909-4d86-9305-720c63771d05-istio-podinfo\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.842297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.836116 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d89558b-7909-4d86-9305-720c63771d05-istiod-ca-cert\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.842297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.836188 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc2dn\" (UniqueName: \"kubernetes.io/projected/6d89558b-7909-4d86-9305-720c63771d05-kube-api-access-xc2dn\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.842297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.751127 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:47:28.842297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.836703 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:47:28.842297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.839975 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-istio-data\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.842297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.841848 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-credential-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.843898 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.843871 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-workload-socket\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.869997 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.845407 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d89558b-7909-4d86-9305-720c63771d05-istiod-ca-cert\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.870563 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.849131 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:28.871075 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.858160 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:47:28.871246 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.871214 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 00:47:28.885793 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.885761 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-istio-envoy\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.901237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.887763 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d89558b-7909-4d86-9305-720c63771d05-istio-podinfo\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.901237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.893588 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc2dn\" (UniqueName: \"kubernetes.io/projected/6d89558b-7909-4d86-9305-720c63771d05-kube-api-access-xc2dn\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.901237 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.895127 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d89558b-7909-4d86-9305-720c63771d05-istio-token\") pod \"e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:28.936882 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:28.936852 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:47:29.152385 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.152290 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:47:29.174618 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.153570 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:29.205913 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.205867 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:47:29.451842 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.434883 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:29.471257 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.471199 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:29.475745 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.475702 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:29.477859 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.477820 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:47:29.478165 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.478078 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:47:29.512665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.504943 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:29.522365 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.513677 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:47:29.523780 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.514359 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:29.543236 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.543208 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:47:29.548863 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.548832 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:47:29.782909 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.782873 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:47:29.783194 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.783159 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:29.814835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.814802 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:29.816380 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.816347 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:47:29.816496 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.816421 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:47:29.833166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.816974 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:47:29.833166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.817016 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 00:47:29.833166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.818555 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:29.847199 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.846886 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:29.859301 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.859274 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:47:29.881654 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.868330 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:29.881654 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.879083 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:47:29.887717 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.887668 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:47:29.887899 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:29.887872 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 00:47:30.069343 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:30.069298 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr"] Oct 31 00:47:30.076676 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:30.076605 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:30.078691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:30.078653 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:30.090814 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:30.090761 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:30.245149 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:30.245118 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" event={"ID":"6d89558b-7909-4d86-9305-720c63771d05","Type":"ContainerStarted","Data":"02d7e4fcdd995cb9c0c352fb01c884bea87c1039fdf48e84493c707133ceccf5"} Oct 31 00:47:30.273678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:30.271902 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:30.293839 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:30.289063 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:47:30.293839 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:30.290898 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:47:30.459658 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:30.458435 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded" Oct 31 00:47:30.486661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:30.477739 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:30.696712 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:30.683964 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:47:30.696712 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:30.684024 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 00:47:30.891660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:30.885384 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:30.891660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:30.887524 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:47:30.983711 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:30.981846 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:31.044717 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:31.043172 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:31.140661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:31.139245 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:31.145066 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:31.143103 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:47:31.260712 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:31.258394 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" event={"ID":"6d89558b-7909-4d86-9305-720c63771d05","Type":"ContainerStarted","Data":"6e38faa53ef9860a61bfaf406e80af578c7f05723c01b146004605bf1c0847f9"} Oct 31 00:47:31.260712 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:31.259746 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:47:31.373297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:31.373223 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" podStartSLOduration=3.373198556 podStartE2EDuration="3.373198556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:31.337460139 +0000 UTC m=+847.366640531" watchObservedRunningTime="2026-10-31 00:47:31.373198556 +0000 UTC m=+847.402378946" Oct 31 00:47:31.810408 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:31.810376 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:31.812108 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:31.812076 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:47:32.059663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:32.059596 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:32.163712 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:32.158720 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:32.163712 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:32.159712 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:47:32.163712 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:32.159771 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" podUID="6d89558b-7909-4d86-9305-720c63771d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 00:47:32.877470 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:32.877426 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:33.154112 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:33.153969 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:47:33.154112 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:33.154078 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" podUID="6d89558b-7909-4d86-9305-720c63771d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 00:47:33.694707 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:33.694671 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:34.154501 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:34.154468 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:47:34.154923 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:34.154519 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" podUID="6d89558b-7909-4d86-9305-720c63771d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 00:47:34.541662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:34.522884 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:35.154712 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:35.154675 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:47:35.155179 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:35.154739 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" podUID="6d89558b-7909-4d86-9305-720c63771d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 00:47:35.328156 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:35.328120 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:36.057074 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:36.056998 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:36.157555 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:36.157492 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:47:36.158461 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:36.158302 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" podUID="6d89558b-7909-4d86-9305-720c63771d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 00:47:36.838573 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:36.838534 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:36.934876 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:36.934840 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:37.003432 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.003373 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd"] Oct 31 00:47:37.010460 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.010431 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" Oct 31 00:47:37.014484 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.014452 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd"] Oct 31 00:47:37.028736 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.023247 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:47:37.028736 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.023321 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:47:37.096194 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.096114 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd\" (UID: \"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" Oct 31 00:47:37.096194 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.096190 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7qlp\" (UniqueName: \"kubernetes.io/projected/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-kube-api-access-t7qlp\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd\" (UID: \"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" Oct 31 00:47:37.096434 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.096217 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd\" (UID: \"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" Oct 31 00:47:37.155054 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.154994 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:47:37.155249 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.155100 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" podUID="6d89558b-7909-4d86-9305-720c63771d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 00:47:37.197702 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.197672 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd\" (UID: \"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" Oct 31 00:47:37.198083 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.197744 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7qlp\" (UniqueName: \"kubernetes.io/projected/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-kube-api-access-t7qlp\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd\" (UID: \"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" Oct 31 00:47:37.198083 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.197771 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd\" (UID: \"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" Oct 31 00:47:37.198502 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.198465 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd\" (UID: \"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" Oct 31 00:47:37.200116 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.200094 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd\" (UID: \"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" Oct 31 00:47:37.206413 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.206390 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7qlp\" (UniqueName: \"kubernetes.io/projected/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-kube-api-access-t7qlp\") pod \"maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd\" (UID: \"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" Oct 31 00:47:37.385784 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.385692 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" Oct 31 00:47:37.668523 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:47:37.668491 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99cc99c_4c6c_45b8_b96a_a45d78e9c5c9.slice/crio-ffadaf2ab46c477e591d82fb64f4458f1b7db5be9af31092348ded3af6abf75c WatchSource:0}: Error finding container ffadaf2ab46c477e591d82fb64f4458f1b7db5be9af31092348ded3af6abf75c: Status 404 returned error can't find the container with id ffadaf2ab46c477e591d82fb64f4458f1b7db5be9af31092348ded3af6abf75c Oct 31 00:47:37.669188 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.669159 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd"] Oct 31 00:47:37.838318 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:37.838286 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:38.155371 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:38.155329 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:47:38.155544 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:38.155406 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" podUID="6d89558b-7909-4d86-9305-720c63771d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 00:47:38.380843 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:38.380753 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" event={"ID":"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9","Type":"ContainerStarted","Data":"b818b342ff3e6653b1abfd7a1dfc5119472d7e39eeaa8c7c2feececf79a97799"} Oct 31 00:47:38.380843 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:38.380825 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" event={"ID":"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9","Type":"ContainerStarted","Data":"ffadaf2ab46c477e591d82fb64f4458f1b7db5be9af31092348ded3af6abf75c"} Oct 31 00:47:38.398465 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:38.398412 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" podStartSLOduration=2.398398381 podStartE2EDuration="2.398398381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:38.396961864 +0000 UTC m=+854.426142276" watchObservedRunningTime="2026-10-31 00:47:38.398398381 +0000 UTC m=+854.427578771" Oct 31 00:47:38.563614 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:38.563563 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:39.158869 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:39.158840 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:39.159213 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:39.159194 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:39.159968 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:39.159954 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:47:39.294508 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:39.294480 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:39.388790 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:39.388759 4519 generic.go:356] "Generic (PLEG): container finished" podID="a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9" containerID="b818b342ff3e6653b1abfd7a1dfc5119472d7e39eeaa8c7c2feececf79a97799" exitCode=0 Oct 31 00:47:39.389288 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:39.388849 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" event={"ID":"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9","Type":"ContainerDied","Data":"b818b342ff3e6653b1abfd7a1dfc5119472d7e39eeaa8c7c2feececf79a97799"} Oct 31 00:47:40.026405 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:40.026370 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:40.568143 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:40.568114 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" Oct 31 00:47:40.633544 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:40.633508 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-maas-api-openshift-ca\") pod \"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9\" (UID: \"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9\") " Oct 31 00:47:40.633753 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:40.633573 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-maas-api-xks-ca\") pod \"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9\" (UID: \"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9\") " Oct 31 00:47:40.633753 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:40.633607 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-kube-api-access-t7qlp\" (UniqueName: \"kubernetes.io/projected/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-kube-api-access-t7qlp\") pod \"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9\" (UID: \"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9\") " Oct 31 00:47:40.634010 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:40.633977 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-maas-api-openshift-ca" pod "a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9" (UID: "a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:40.634531 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:40.634506 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:40.636352 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:40.636325 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-kube-api-access-t7qlp" pod "a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9" (UID: "a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9"). InnerVolumeSpecName "kube-api-access-t7qlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:40.636457 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:40.636323 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-maas-api-xks-ca" pod "a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9" (UID: "a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:40.735854 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:40.735827 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:40.735854 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:40.735854 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7qlp\" (UniqueName: \"kubernetes.io/projected/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9-kube-api-access-t7qlp\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:40.903913 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:40.903877 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:41.402369 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:41.402335 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" Oct 31 00:47:41.402522 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:41.402368 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd" event={"ID":"a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9","Type":"ContainerDied","Data":"ffadaf2ab46c477e591d82fb64f4458f1b7db5be9af31092348ded3af6abf75c"} Oct 31 00:47:41.402522 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:41.402418 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffadaf2ab46c477e591d82fb64f4458f1b7db5be9af31092348ded3af6abf75c" Oct 31 00:47:41.781879 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:41.781848 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:41.891093 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:41.891054 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg"] Oct 31 00:47:41.985892 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:41.985861 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:42.756257 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:42.756220 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:43.851045 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:43.850953 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:44.149709 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:44.147917 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded" Oct 31 00:47:44.187894 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:44.187852 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:44.295755 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:44.295714 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:44.426719 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:44.423942 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:47:44.574873 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:44.574816 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:44.679191 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:44.679062 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:44.716686 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:44.715789 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:44.805229 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:44.805186 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:44.834844 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:44.834778 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:44.882672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:44.880688 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 00:47:45.099538 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.099387 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:47:45.221911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.221868 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:45.223868 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.222735 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:47:45.225062 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.225033 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:45.318964 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.318905 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:45.436046 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.432764 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:47:45.436046 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.433256 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:47:45.436046 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.433319 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:47:45.436046 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.435923 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:45.452176 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.452132 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 31 00:47:45.456500 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.456466 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:45.803405 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.534130 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:47:45.838859 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.838801 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 00:47:45.859921 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.856746 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:47:45.901360 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.901307 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 00:47:45.902415 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.902393 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:47:45.922337 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.922284 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:47:45.922622 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:45.922577 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:46.027674 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:46.023781 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:47:46.162691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:46.162575 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:47:46.281797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:46.281747 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 00:47:46.381662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:46.372278 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:46.497157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:46.454066 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:46.497157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:46.457080 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:46.497157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:46.462526 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:46.633416 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:46.632777 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:47.035661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.027965 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:47.079925 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.079873 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:47.112402 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.112218 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:47:47.112402 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.112284 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:47.146813 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.146749 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:47.170671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.163101 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:47.199159 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.199052 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:47:47.199159 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.199114 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:47.293710 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.288864 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:47.324328 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.318746 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:47.420025 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.419967 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:47.472868 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.452570 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:47.481786 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.458781 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 00:47:47.508774 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.458917 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:47:47.605966 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.602890 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:47.631775 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:47:47.615545 4519 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.409s" Oct 31 00:47:47.631775 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.615714 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:47:47.631775 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.615754 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:47:47.631775 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.615780 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs"] Oct 31 00:47:47.631775 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.615831 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:47:47.631775 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.615864 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:47:47.631775 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.615892 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:47:47.631775 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.618885 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:47:47.634863 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.634825 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:47:47.639814 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.639782 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:47:47.728828 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.688098 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:47:47.814450 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.814289 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:47:47.825875 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.825847 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:47:47.911742 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:47.908630 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:47.955097 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:47:47.955055 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:47:48.005483 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:48.005401 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:47:48.657876 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:48.657808 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:49.103344 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:49.103307 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:50.089881 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:50.089854 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:50.852623 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:50.852589 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:51.814155 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:51.814117 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:52.471413 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:52.471377 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:52.623815 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:52.623774 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" podUID="d39d45bb-7e82-4c44-8038-491cf125e145" containerName="payload-pre-processing" containerID="cri-o://985f3bfd29a7b8f1705cfb0b397b347962bdf17500f388c8ecf5f3fae78123cc" gracePeriod=25 Oct 31 00:47:53.127252 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:53.127213 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:53.660271 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:53.660233 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz"] Oct 31 00:47:53.712679 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:53.712613 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:53.886900 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:53.886868 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:54.502445 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:54.502404 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" podUID="d39d45bb-7e82-4c44-8038-491cf125e145" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:47:54.834106 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:54.834022 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:55.471429 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.471383 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:55.571749 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.571717 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:55.663383 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.662534 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln"] Oct 31 00:47:55.663585 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.663478 4519 cpu_manager.go:414] "containerMap: removing container" podUID="a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9" containerName="revoke-keys" Oct 31 00:47:55.663585 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.663502 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9" containerName="revoke-keys" Oct 31 00:47:55.663732 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.663724 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9" containerName="revoke-keys" Oct 31 00:47:55.687508 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.684335 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln" Oct 31 00:47:55.714434 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.714407 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:47:55.715608 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.715578 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:47:55.772152 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.772123 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln"] Oct 31 00:47:55.790898 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.790864 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpt7n\" (UniqueName: \"kubernetes.io/projected/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-kube-api-access-xpt7n\") pod \"maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln\" (UID: \"8d7a5f83-6b68-484a-a30f-9f59eca82c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln" Oct 31 00:47:55.791218 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.791187 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln\" (UID: \"8d7a5f83-6b68-484a-a30f-9f59eca82c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln" Oct 31 00:47:55.791562 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.791529 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln\" (UID: \"8d7a5f83-6b68-484a-a30f-9f59eca82c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln" Oct 31 00:47:55.892553 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.892511 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln\" (UID: \"8d7a5f83-6b68-484a-a30f-9f59eca82c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln" Oct 31 00:47:55.892762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.892571 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln\" (UID: \"8d7a5f83-6b68-484a-a30f-9f59eca82c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln" Oct 31 00:47:55.892762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.892613 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpt7n\" (UniqueName: \"kubernetes.io/projected/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-kube-api-access-xpt7n\") pod \"maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln\" (UID: \"8d7a5f83-6b68-484a-a30f-9f59eca82c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln" Oct 31 00:47:55.893340 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.893314 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln\" (UID: \"8d7a5f83-6b68-484a-a30f-9f59eca82c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln" Oct 31 00:47:55.895101 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.895081 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln\" (UID: \"8d7a5f83-6b68-484a-a30f-9f59eca82c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln" Oct 31 00:47:55.901905 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:55.901875 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpt7n\" (UniqueName: \"kubernetes.io/projected/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-kube-api-access-xpt7n\") pod \"maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln\" (UID: \"8d7a5f83-6b68-484a-a30f-9f59eca82c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln" Oct 31 00:47:56.076424 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:56.076344 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln" Oct 31 00:47:56.362756 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:56.362722 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln"] Oct 31 00:47:56.363427 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:47:56.363398 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7a5f83_6b68_484a_a30f_9f59eca82c4e.slice/crio-15568d53196e3c3fa684835fa9c25b94d17a9a29ec1e3cefb58317e746198f90 WatchSource:0}: Error finding container 15568d53196e3c3fa684835fa9c25b94d17a9a29ec1e3cefb58317e746198f90: Status 404 returned error can't find the container with id 15568d53196e3c3fa684835fa9c25b94d17a9a29ec1e3cefb58317e746198f90 Oct 31 00:47:56.512869 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:56.512838 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:56.812659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:56.803723 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln" event={"ID":"8d7a5f83-6b68-484a-a30f-9f59eca82c4e","Type":"ContainerStarted","Data":"15568d53196e3c3fa684835fa9c25b94d17a9a29ec1e3cefb58317e746198f90"} Oct 31 00:47:57.000682 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:57.000567 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" containerID="cri-o://75442217dff62ea366113494e6eb83bf9b5063062802f679b55cee31eeb1cd67" gracePeriod=45 Oct 31 00:47:57.518733 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:57.516479 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:58.212074 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.179873 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:47:58.212074 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.204033 4519 generic.go:356] "Generic (PLEG): container finished" podID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerID="75442217dff62ea366113494e6eb83bf9b5063062802f679b55cee31eeb1cd67" exitCode=0 Oct 31 00:47:58.281670 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.257447 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" event={"ID":"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc","Type":"ContainerDied","Data":"75442217dff62ea366113494e6eb83bf9b5063062802f679b55cee31eeb1cd67"} Oct 31 00:47:58.364661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.364544 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln" event={"ID":"8d7a5f83-6b68-484a-a30f-9f59eca82c4e","Type":"ContainerStarted","Data":"6c566468af90ba3a6a293880cd06ccc55cd311be9e6b9712b6d5a375333b18aa"} Oct 31 00:47:58.364938 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.318127 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:47:58.365401 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.316862 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:58.413840 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.413788 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:58.603667 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.596782 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:58.641718 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.638202 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-model-cache\") pod \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " Oct 31 00:47:58.689925 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.689891 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-dshm\") pod \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " Oct 31 00:47:58.690295 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.690271 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-kube-api-access-z2h9w\" (UniqueName: \"kubernetes.io/projected/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-kube-api-access-z2h9w\") pod \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " Oct 31 00:47:58.690855 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.690833 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-home\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-home\") pod \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " Oct 31 00:47:58.691149 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.691131 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-tmp-dir\") pod \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " Oct 31 00:47:58.691279 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.691265 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-tls-certs\" (UniqueName: \"kubernetes.io/secret/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-tls-certs\") pod \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\" (UID: \"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc\") " Oct 31 00:47:58.695831 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.657863 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-model-cache" pod "780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" (UID: "780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:58.708681 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.703184 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-home" pod "780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" (UID: "780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:58.736667 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.713141 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:58.737007 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.736971 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-tmp-dir" pod "780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" (UID: "780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:58.784852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.784806 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-tls-certs" pod "780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" (UID: "780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:58.812053 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.800193 4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-tmp-dir\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:58.812322 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.812302 4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-model-cache\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:58.812433 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.812419 4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-home\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:58.838956 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.838186 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 00:47:58.845904 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.839734 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-dshm" pod "780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" (UID: "780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:58.846414 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.846343 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:58.868959 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.868901 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-kube-api-access-z2h9w" pod "780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" (UID: "780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc"). InnerVolumeSpecName "kube-api-access-z2h9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:58.901851 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.901732 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 00:47:58.920669 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.913956 4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-tls-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:58.920987 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.920957 4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-dshm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:58.921121 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:58.921105 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2h9w\" (UniqueName: \"kubernetes.io/projected/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc-kube-api-access-z2h9w\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:59.017795 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.017745 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:47:59.087780 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.087725 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:59.295710 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.288749 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:47:59.295710 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.289229 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:47:59.382662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.378275 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:59.382662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.379070 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:47:59.419808 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.419764 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:59.420545 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.420518 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:47:59.617070 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.616974 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:59.623667 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.617795 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:47:59.677665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.655727 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded" Oct 31 00:47:59.677665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.655776 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:59.727661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.722184 4519 generic.go:356] "Generic (PLEG): container finished" podID="8d7a5f83-6b68-484a-a30f-9f59eca82c4e" containerID="6c566468af90ba3a6a293880cd06ccc55cd311be9e6b9712b6d5a375333b18aa" exitCode=0 Oct 31 00:47:59.749660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.737678 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln" event={"ID":"8d7a5f83-6b68-484a-a30f-9f59eca82c4e","Type":"ContainerDied","Data":"6c566468af90ba3a6a293880cd06ccc55cd311be9e6b9712b6d5a375333b18aa"} Oct 31 00:47:59.752142 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.752097 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" event={"ID":"780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc","Type":"ContainerDied","Data":"189197a0b832c37133b573880f57ab2662000b6ee2943b0cc827dad25ff4a18a"} Oct 31 00:47:59.752300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.752167 4519 scope.go:122] "RemoveContainer" containerID="75442217dff62ea366113494e6eb83bf9b5063062802f679b55cee31eeb1cd67" Oct 31 00:47:59.752410 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.752391 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg" Oct 31 00:47:59.851746 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.851689 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:59.852412 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.852033 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:59.875359 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.852864 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:47:59.888801 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.888756 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:47:59.933829 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.905122 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:59.933829 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:47:59.905909 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:48:00.040444 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:00.040400 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:48:00.041189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:00.041166 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:48:00.077949 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:00.077899 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:00.134864 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:00.118990 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:48:00.305732 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:00.291884 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:48:00.411418 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:00.405944 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 31 00:48:00.423764 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:00.422852 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:48:00.477662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:00.468541 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:48:00.543683 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:00.542708 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:00.670739 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:00.658033 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:00.672841 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:00.672731 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:00.886366 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:00.884834 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 00:48:00.922780 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:00.906173 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:00.925709 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:00.925426 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:48:00.942789 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:00.942743 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 00:48:00.984129 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:00.984089 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:48:00.984381 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:00.984351 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:01.045366 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:01.045318 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 00:48:01.395762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:01.058744 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:01.431799 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:01.059201 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:01.431799 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:01.059230 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded" Oct 31 00:48:01.431799 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:01.092910 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:01.431799 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:01.204238 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:48:01.431799 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:01.429400 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:48:01.431799 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:01.407091 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:01.431799 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:01.325038 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:48:01.431799 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:01.423740 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:48:01.452304 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:01.452266 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:01.598057 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:01.598010 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:01.685793 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:01.683668 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 31 00:48:01.685793 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:01.683784 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 00:48:01.738887 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:01.738850 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg"] Oct 31 00:48:01.898662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:01.890145 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 00:48:01.960853 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:01.960719 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:02.092384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.092344 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1d4a7e/distinct-2-w3-1d4a7e-kserve-7c4ffc9dc5-sz6jg"] Oct 31 00:48:02.280661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.251947 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:48:02.280661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.252015 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:02.280661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.252842 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" path="/var/lib/kubelet/pods/780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc/volumes" Oct 31 00:48:02.280661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.253310 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx"] Oct 31 00:48:02.280661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.253353 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp"] Oct 31 00:48:02.304813 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.253739 4519 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:48:02.325659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.318953 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj"] Oct 31 00:48:02.325659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.319073 4519 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:48:02.358449 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.358282 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:48:02.358449 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.358350 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:02.422684 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.417250 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 00:48:02.470290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.431455 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc"] Oct 31 00:48:02.470290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.432048 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:02.470290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.432396 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:48:02.470290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.432699 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:48:02.470290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.432953 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:02.470290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.433217 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:48:02.470290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.433488 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 31 00:48:02.470290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.433778 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:48:02.520557 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.520511 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:02.687839 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.687730 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:02.891997 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.890942 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:02.920727 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.905822 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:48:02.920727 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:02.905889 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:03.021695 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.004421 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 00:48:03.250342 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.250187 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:03.397171 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.397025 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln" Oct 31 00:48:03.423787 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.420599 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:03.507907 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.488387 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:48:03.507907 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.492712 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:03.517565 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.510904 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:03.524680 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.521734 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:48:03.524680 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.523035 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 00:48:03.524680 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.524204 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:48:03.561336 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.526293 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:48:03.585971 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.583201 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-kube-api-access-xpt7n\" (UniqueName: \"kubernetes.io/projected/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-kube-api-access-xpt7n\") pod \"8d7a5f83-6b68-484a-a30f-9f59eca82c4e\" (UID: \"8d7a5f83-6b68-484a-a30f-9f59eca82c4e\") " Oct 31 00:48:03.585971 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.583382 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-maas-api-openshift-ca\") pod \"8d7a5f83-6b68-484a-a30f-9f59eca82c4e\" (UID: \"8d7a5f83-6b68-484a-a30f-9f59eca82c4e\") " Oct 31 00:48:03.585971 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.583566 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-maas-api-xks-ca\") pod \"8d7a5f83-6b68-484a-a30f-9f59eca82c4e\" (UID: \"8d7a5f83-6b68-484a-a30f-9f59eca82c4e\") " Oct 31 00:48:03.605123 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.603446 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-maas-api-openshift-ca" pod "8d7a5f83-6b68-484a-a30f-9f59eca82c4e" (UID: "8d7a5f83-6b68-484a-a30f-9f59eca82c4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:03.649198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.648779 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln" event={"ID":"8d7a5f83-6b68-484a-a30f-9f59eca82c4e","Type":"ContainerDied","Data":"15568d53196e3c3fa684835fa9c25b94d17a9a29ec1e3cefb58317e746198f90"} Oct 31 00:48:03.649198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.648843 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15568d53196e3c3fa684835fa9c25b94d17a9a29ec1e3cefb58317e746198f90" Oct 31 00:48:03.649198 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.648914 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln" Oct 31 00:48:03.692670 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.690881 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:03.692670 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.690968 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:03.804868 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.801795 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-maas-api-xks-ca" pod "8d7a5f83-6b68-484a-a30f-9f59eca82c4e" (UID: "8d7a5f83-6b68-484a-a30f-9f59eca82c4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:03.846999 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.846454 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-kube-api-access-xpt7n" pod "8d7a5f83-6b68-484a-a30f-9f59eca82c4e" (UID: "8d7a5f83-6b68-484a-a30f-9f59eca82c4e"). InnerVolumeSpecName "kube-api-access-xpt7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:03.899936 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.899509 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:03.899936 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.899545 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpt7n\" (UniqueName: \"kubernetes.io/projected/8d7a5f83-6b68-484a-a30f-9f59eca82c4e-kube-api-access-xpt7n\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:03.899936 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.899788 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:03.901575 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.900438 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:03.929695 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:03.929373 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:48:04.192733 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.084386 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:04.192733 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.105805 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:04.481589 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.480650 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:04.528602 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.492751 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:48:04.528602 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.492902 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:48:04.528602 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.506990 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" podUID="d39d45bb-7e82-4c44-8038-491cf125e145" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:48:04.528602 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.523414 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:04.528602 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.523794 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:04.528602 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.524419 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:04.528602 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.526446 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:04.556691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.543502 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7"] Oct 31 00:48:04.556691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.546915 4519 cpu_manager.go:414] "containerMap: removing container" podUID="8d7a5f83-6b68-484a-a30f-9f59eca82c4e" containerName="revoke-keys" Oct 31 00:48:04.556691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.546940 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7a5f83-6b68-484a-a30f-9f59eca82c4e" containerName="revoke-keys" Oct 31 00:48:04.556691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.546959 4519 cpu_manager.go:414] "containerMap: removing container" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" Oct 31 00:48:04.556691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.546969 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" Oct 31 00:48:04.556691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.547166 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7a5f83-6b68-484a-a30f-9f59eca82c4e" containerName="revoke-keys" Oct 31 00:48:04.556691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.547180 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="780f7e8a-9e3b-40e2-b266-a3c03ab5bdcc" containerName="main" Oct 31 00:48:04.569628 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.560039 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" Oct 31 00:48:04.579722 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.573116 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:48:04.581669 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.581629 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:04.585328 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.585305 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:48:04.618460 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.615935 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:48:04.693819 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.692271 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded" Oct 31 00:48:04.785550 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.785456 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7"] Oct 31 00:48:04.825405 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.825364 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:48:04.852228 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.843678 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 00:48:04.852228 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.846188 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:48:04.937500 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.935943 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:48:04.939761 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.939114 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e0154e-7cfa-43d1-b996-65611ea1ef4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7\" (UID: \"07e0154e-7cfa-43d1-b996-65611ea1ef4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" Oct 31 00:48:04.939761 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.939164 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcdtq\" (UniqueName: \"kubernetes.io/projected/07e0154e-7cfa-43d1-b996-65611ea1ef4e-kube-api-access-rcdtq\") pod \"maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7\" (UID: \"07e0154e-7cfa-43d1-b996-65611ea1ef4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" Oct 31 00:48:04.939761 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.939304 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e0154e-7cfa-43d1-b996-65611ea1ef4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7\" (UID: \"07e0154e-7cfa-43d1-b996-65611ea1ef4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" Oct 31 00:48:04.944647 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.944601 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:48:04.948043 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.947927 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:48:04.971479 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.971447 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:48:04.971768 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.971750 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:48:04.975337 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:04.975317 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:48:05.041711 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:05.041608 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e0154e-7cfa-43d1-b996-65611ea1ef4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7\" (UID: \"07e0154e-7cfa-43d1-b996-65611ea1ef4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" Oct 31 00:48:05.042824 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:05.042779 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e0154e-7cfa-43d1-b996-65611ea1ef4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7\" (UID: \"07e0154e-7cfa-43d1-b996-65611ea1ef4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" Oct 31 00:48:05.043001 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:05.041956 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcdtq\" (UniqueName: \"kubernetes.io/projected/07e0154e-7cfa-43d1-b996-65611ea1ef4e-kube-api-access-rcdtq\") pod \"maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7\" (UID: \"07e0154e-7cfa-43d1-b996-65611ea1ef4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" Oct 31 00:48:05.043252 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:05.043234 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e0154e-7cfa-43d1-b996-65611ea1ef4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7\" (UID: \"07e0154e-7cfa-43d1-b996-65611ea1ef4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" Oct 31 00:48:05.052612 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:05.052572 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e0154e-7cfa-43d1-b996-65611ea1ef4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7\" (UID: \"07e0154e-7cfa-43d1-b996-65611ea1ef4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" Oct 31 00:48:05.056502 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:05.056475 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcdtq\" (UniqueName: \"kubernetes.io/projected/07e0154e-7cfa-43d1-b996-65611ea1ef4e-kube-api-access-rcdtq\") pod \"maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7\" (UID: \"07e0154e-7cfa-43d1-b996-65611ea1ef4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" Oct 31 00:48:05.163995 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:05.163961 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" Oct 31 00:48:05.457581 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:05.457550 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7"] Oct 31 00:48:05.458276 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:48:05.458250 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e0154e_7cfa_43d1_b996_65611ea1ef4e.slice/crio-753936c86dd2aacbcdb478700c3589871e4fba5847fdb87f8c82d2bd4f6a203f WatchSource:0}: Error finding container 753936c86dd2aacbcdb478700c3589871e4fba5847fdb87f8c82d2bd4f6a203f: Status 404 returned error can't find the container with id 753936c86dd2aacbcdb478700c3589871e4fba5847fdb87f8c82d2bd4f6a203f Oct 31 00:48:05.819540 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:05.819501 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:05.870771 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:05.870739 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp"] Oct 31 00:48:05.875582 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:05.875549 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv"] Oct 31 00:48:05.901254 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:05.901214 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" event={"ID":"07e0154e-7cfa-43d1-b996-65611ea1ef4e","Type":"ContainerStarted","Data":"f02c07d768765b6e6b5ec0e565cd629843324ce035c36d623447af55c514a6f9"} Oct 31 00:48:05.901458 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:05.901280 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" event={"ID":"07e0154e-7cfa-43d1-b996-65611ea1ef4e","Type":"ContainerStarted","Data":"753936c86dd2aacbcdb478700c3589871e4fba5847fdb87f8c82d2bd4f6a203f"} Oct 31 00:48:05.913943 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:05.913914 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:05.931946 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:05.931891 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" podStartSLOduration=1.931879517 podStartE2EDuration="1.931879517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:05.931853124 +0000 UTC m=+881.961033516" watchObservedRunningTime="2026-10-31 00:48:05.931879517 +0000 UTC m=+881.961059906" Oct 31 00:48:05.983047 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:05.982966 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:06.909744 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:06.909711 4519 generic.go:356] "Generic (PLEG): container finished" podID="07e0154e-7cfa-43d1-b996-65611ea1ef4e" containerID="f02c07d768765b6e6b5ec0e565cd629843324ce035c36d623447af55c514a6f9" exitCode=0 Oct 31 00:48:06.910123 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:06.909793 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" event={"ID":"07e0154e-7cfa-43d1-b996-65611ea1ef4e","Type":"ContainerDied","Data":"f02c07d768765b6e6b5ec0e565cd629843324ce035c36d623447af55c514a6f9"} Oct 31 00:48:07.390071 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:07.390020 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:08.083666 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.083604 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" Oct 31 00:48:08.108887 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.108853 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e0154e-7cfa-43d1-b996-65611ea1ef4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e0154e-7cfa-43d1-b996-65611ea1ef4e-maas-api-xks-ca\") pod \"07e0154e-7cfa-43d1-b996-65611ea1ef4e\" (UID: \"07e0154e-7cfa-43d1-b996-65611ea1ef4e\") " Oct 31 00:48:08.109059 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.108951 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e0154e-7cfa-43d1-b996-65611ea1ef4e-kube-api-access-rcdtq\" (UniqueName: \"kubernetes.io/projected/07e0154e-7cfa-43d1-b996-65611ea1ef4e-kube-api-access-rcdtq\") pod \"07e0154e-7cfa-43d1-b996-65611ea1ef4e\" (UID: \"07e0154e-7cfa-43d1-b996-65611ea1ef4e\") " Oct 31 00:48:08.109059 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.109049 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e0154e-7cfa-43d1-b996-65611ea1ef4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e0154e-7cfa-43d1-b996-65611ea1ef4e-maas-api-openshift-ca\") pod \"07e0154e-7cfa-43d1-b996-65611ea1ef4e\" (UID: \"07e0154e-7cfa-43d1-b996-65611ea1ef4e\") " Oct 31 00:48:08.109619 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.109590 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e0154e-7cfa-43d1-b996-65611ea1ef4e-maas-api-openshift-ca" pod "07e0154e-7cfa-43d1-b996-65611ea1ef4e" (UID: "07e0154e-7cfa-43d1-b996-65611ea1ef4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:08.111969 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.111945 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e0154e-7cfa-43d1-b996-65611ea1ef4e-maas-api-xks-ca" pod "07e0154e-7cfa-43d1-b996-65611ea1ef4e" (UID: "07e0154e-7cfa-43d1-b996-65611ea1ef4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:08.112150 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.112123 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e0154e-7cfa-43d1-b996-65611ea1ef4e-kube-api-access-rcdtq" pod "07e0154e-7cfa-43d1-b996-65611ea1ef4e" (UID: "07e0154e-7cfa-43d1-b996-65611ea1ef4e"). InnerVolumeSpecName "kube-api-access-rcdtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:08.211165 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.211081 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e0154e-7cfa-43d1-b996-65611ea1ef4e-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:08.211165 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.211120 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e0154e-7cfa-43d1-b996-65611ea1ef4e-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:08.211165 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.211136 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcdtq\" (UniqueName: \"kubernetes.io/projected/07e0154e-7cfa-43d1-b996-65611ea1ef4e-kube-api-access-rcdtq\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:08.441833 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.441789 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:08.684911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.684873 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" containerID="cri-o://bf44831817b786fce85a1ba53f831f4652fc9c845ea094494105f3262f12cc26" gracePeriod=45 Oct 31 00:48:08.959607 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.959580 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" Oct 31 00:48:08.959786 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.959584 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7" event={"ID":"07e0154e-7cfa-43d1-b996-65611ea1ef4e","Type":"ContainerDied","Data":"753936c86dd2aacbcdb478700c3589871e4fba5847fdb87f8c82d2bd4f6a203f"} Oct 31 00:48:08.959786 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.959699 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="753936c86dd2aacbcdb478700c3589871e4fba5847fdb87f8c82d2bd4f6a203f" Oct 31 00:48:08.960864 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.960840 4519 generic.go:356] "Generic (PLEG): container finished" podID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerID="bf44831817b786fce85a1ba53f831f4652fc9c845ea094494105f3262f12cc26" exitCode=0 Oct 31 00:48:08.960990 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.960896 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" event={"ID":"bf5f1905-6c45-4ea7-8871-5676c27e80a7","Type":"ContainerDied","Data":"bf44831817b786fce85a1ba53f831f4652fc9c845ea094494105f3262f12cc26"} Oct 31 00:48:08.963063 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.963042 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:48:08.981166 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:08.981136 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d"] Oct 31 00:48:09.046517 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.046446 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-model-cache\") pod \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " Oct 31 00:48:09.046721 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.046553 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5f1905-6c45-4ea7-8871-5676c27e80a7-kube-api-access-j2ss5\" (UniqueName: \"kubernetes.io/projected/bf5f1905-6c45-4ea7-8871-5676c27e80a7-kube-api-access-j2ss5\") pod \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " Oct 31 00:48:09.046810 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.046768 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-model-cache" pod "bf5f1905-6c45-4ea7-8871-5676c27e80a7" (UID: "bf5f1905-6c45-4ea7-8871-5676c27e80a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:09.046810 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.046780 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-home\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-home\") pod \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " Oct 31 00:48:09.046916 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.046897 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf5f1905-6c45-4ea7-8871-5676c27e80a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5f1905-6c45-4ea7-8871-5676c27e80a7-tls-certs\") pod \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " Oct 31 00:48:09.046979 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.046966 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-dshm\") pod \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " Oct 31 00:48:09.047031 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.047003 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-tmp-dir\") pod \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\" (UID: \"bf5f1905-6c45-4ea7-8871-5676c27e80a7\") " Oct 31 00:48:09.047089 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.047067 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-home" pod "bf5f1905-6c45-4ea7-8871-5676c27e80a7" (UID: "bf5f1905-6c45-4ea7-8871-5676c27e80a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:09.047628 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.047601 4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-home\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.047764 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.047631 4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-model-cache\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.047929 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.047906 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-tmp-dir" pod "bf5f1905-6c45-4ea7-8871-5676c27e80a7" (UID: "bf5f1905-6c45-4ea7-8871-5676c27e80a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:09.049629 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.049577 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5f1905-6c45-4ea7-8871-5676c27e80a7-kube-api-access-j2ss5" pod "bf5f1905-6c45-4ea7-8871-5676c27e80a7" (UID: "bf5f1905-6c45-4ea7-8871-5676c27e80a7"). InnerVolumeSpecName "kube-api-access-j2ss5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:09.050234 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.050206 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf5f1905-6c45-4ea7-8871-5676c27e80a7-tls-certs" pod "bf5f1905-6c45-4ea7-8871-5676c27e80a7" (UID: "bf5f1905-6c45-4ea7-8871-5676c27e80a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:09.050331 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.050302 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-dshm" pod "bf5f1905-6c45-4ea7-8871-5676c27e80a7" (UID: "bf5f1905-6c45-4ea7-8871-5676c27e80a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:09.056318 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.056235 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:09.149377 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.149333 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2ss5\" (UniqueName: \"kubernetes.io/projected/bf5f1905-6c45-4ea7-8871-5676c27e80a7-kube-api-access-j2ss5\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.149377 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.149368 4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5f1905-6c45-4ea7-8871-5676c27e80a7-tls-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.149377 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.149378 4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-dshm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.149831 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.149405 4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5f1905-6c45-4ea7-8871-5676c27e80a7-tmp-dir\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.606850 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.606817 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:09.968887 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.968802 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" Oct 31 00:48:09.968887 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.968853 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz" event={"ID":"bf5f1905-6c45-4ea7-8871-5676c27e80a7","Type":"ContainerDied","Data":"4643e02c283c90850b676f1d3d60d15d3444f3579746492672c79a71b83e7224"} Oct 31 00:48:09.969123 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.968906 4519 scope.go:122] "RemoveContainer" containerID="bf44831817b786fce85a1ba53f831f4652fc9c845ea094494105f3262f12cc26" Oct 31 00:48:09.994672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.994619 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz"] Oct 31 00:48:09.999365 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:09.999339 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-47f370/simulator-w6-47f370-kserve-7895c56c66-nr5jz"] Oct 31 00:48:10.103522 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:10.103492 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" path="/var/lib/kubelet/pods/bf5f1905-6c45-4ea7-8871-5676c27e80a7/volumes" Oct 31 00:48:10.333276 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:10.333236 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:10.871438 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:10.871397 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" podUID="fd5bf7df-a3e9-4de3-ac84-e258676c6f92" containerName="payload-pre-processing" containerID="cri-o://e775d68e1dcfbef4cab975ccba7835d1894fed65283422e6ce56825a368ec5e0" gracePeriod=25 Oct 31 00:48:10.875980 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:10.875946 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" podUID="7dd65b23-b72a-4eb4-971f-5f218554d0f2" containerName="payload-processing" containerID="cri-o://8134f364768d468c6959d1f9c3bf701e1f827eb2c4c57268a3872f0688d0e76e" gracePeriod=25 Oct 31 00:48:11.081252 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:11.081215 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:11.345296 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:11.345265 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:12.047001 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:12.046966 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:12.541456 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:12.541413 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:12.794819 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:12.785192 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:12.928663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:12.925063 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:13.040095 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.022035 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:13.140967 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.136145 4519 generic.go:356] "Generic (PLEG): container finished" podID="d39d45bb-7e82-4c44-8038-491cf125e145" containerID="985f3bfd29a7b8f1705cfb0b397b347962bdf17500f388c8ecf5f3fae78123cc" exitCode=0 Oct 31 00:48:13.140967 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.136233 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" event={"ID":"d39d45bb-7e82-4c44-8038-491cf125e145","Type":"ContainerDied","Data":"985f3bfd29a7b8f1705cfb0b397b347962bdf17500f388c8ecf5f3fae78123cc"} Oct 31 00:48:13.323690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.317653 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d"] Oct 31 00:48:13.323690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.318951 4519 cpu_manager.go:414] "containerMap: removing container" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" Oct 31 00:48:13.323690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.318975 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" Oct 31 00:48:13.323690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.318992 4519 cpu_manager.go:414] "containerMap: removing container" podUID="07e0154e-7cfa-43d1-b996-65611ea1ef4e" containerName="revoke-keys" Oct 31 00:48:13.323690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.318999 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e0154e-7cfa-43d1-b996-65611ea1ef4e" containerName="revoke-keys" Oct 31 00:48:13.323690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.319184 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5f1905-6c45-4ea7-8871-5676c27e80a7" containerName="main" Oct 31 00:48:13.323690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.319200 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e0154e-7cfa-43d1-b996-65611ea1ef4e" containerName="revoke-keys" Oct 31 00:48:13.370664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.343764 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d"] Oct 31 00:48:13.370664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.344834 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.400656 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.400534 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" Oct 31 00:48:13.423780 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.423131 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpnx5\" (UniqueName: \"kubernetes.io/projected/9e7a3b29-178b-43e4-b127-314c2b183d9e-kube-api-access-xpnx5\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.423780 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.423286 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-envoy\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.423780 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.423332 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7a3b29-178b-43e4-b127-314c2b183d9e-istiod-ca-cert\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.423780 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.423457 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-token\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.423780 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.423490 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-workload-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.423780 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.423526 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-credential-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.423780 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.423592 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-workload-certs\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.423780 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.423620 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-podinfo\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.424168 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.423844 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-data\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.446655 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.444430 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:13.524938 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.524907 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d39d45bb-7e82-4c44-8038-491cf125e145-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39d45bb-7e82-4c44-8038-491cf125e145-plugins-config-volume\") pod \"d39d45bb-7e82-4c44-8038-491cf125e145\" (UID: \"d39d45bb-7e82-4c44-8038-491cf125e145\") " Oct 31 00:48:13.525114 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.525047 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39d45bb-7e82-4c44-8038-491cf125e145-kube-api-access-fngq6\" (UniqueName: \"kubernetes.io/projected/d39d45bb-7e82-4c44-8038-491cf125e145-kube-api-access-fngq6\") pod \"d39d45bb-7e82-4c44-8038-491cf125e145\" (UID: \"d39d45bb-7e82-4c44-8038-491cf125e145\") " Oct 31 00:48:13.525375 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.525350 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d39d45bb-7e82-4c44-8038-491cf125e145-plugins-config-volume" pod "d39d45bb-7e82-4c44-8038-491cf125e145" (UID: "d39d45bb-7e82-4c44-8038-491cf125e145"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:13.525486 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.525354 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-token\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.525486 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.525432 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-workload-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.525486 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.525472 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-credential-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.525716 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.525547 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-workload-certs\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.525716 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.525579 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-podinfo\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.525845 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.525731 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-data\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.525845 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.525793 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpnx5\" (UniqueName: \"kubernetes.io/projected/9e7a3b29-178b-43e4-b127-314c2b183d9e-kube-api-access-xpnx5\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.525845 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.525832 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-workload-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.525990 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.525900 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-envoy\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.525990 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.525936 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-workload-certs\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.525990 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.525945 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7a3b29-178b-43e4-b127-314c2b183d9e-istiod-ca-cert\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.526138 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.525902 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-credential-socket\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.526182 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.526118 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-data\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.529188 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.527129 4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39d45bb-7e82-4c44-8038-491cf125e145-plugins-config-volume\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:13.529188 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.527367 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7a3b29-178b-43e4-b127-314c2b183d9e-istiod-ca-cert\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.529188 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.527726 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39d45bb-7e82-4c44-8038-491cf125e145-kube-api-access-fngq6" pod "d39d45bb-7e82-4c44-8038-491cf125e145" (UID: "d39d45bb-7e82-4c44-8038-491cf125e145"). InnerVolumeSpecName "kube-api-access-fngq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:13.533907 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.530942 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-podinfo\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.538276 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.538247 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-envoy\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.540946 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.540920 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpnx5\" (UniqueName: \"kubernetes.io/projected/9e7a3b29-178b-43e4-b127-314c2b183d9e-kube-api-access-xpnx5\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.541328 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.541305 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-token\") pod \"e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.628600 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.628562 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fngq6\" (UniqueName: \"kubernetes.io/projected/d39d45bb-7e82-4c44-8038-491cf125e145-kube-api-access-fngq6\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:13.699007 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.698922 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:13.956739 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.956627 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:13.986695 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.986662 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d"] Oct 31 00:48:13.988162 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:48:13.988137 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7a3b29_178b_43e4_b127_314c2b183d9e.slice/crio-363b3e257dbdad72860ea518e1517a2f1958b2d68128f7d2fc68e7944254e04a WatchSource:0}: Error finding container 363b3e257dbdad72860ea518e1517a2f1958b2d68128f7d2fc68e7944254e04a: Status 404 returned error can't find the container with id 363b3e257dbdad72860ea518e1517a2f1958b2d68128f7d2fc68e7944254e04a Oct 31 00:48:13.991019 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.990980 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:48:13.991109 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.991067 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:48:13.991150 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:13.991109 4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:48:14.167481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:14.167445 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" event={"ID":"d39d45bb-7e82-4c44-8038-491cf125e145","Type":"ContainerDied","Data":"2305eb4f24f97b299d9dafe7bb405f31100e69e8cd30630e9c2f868b05f979fd"} Oct 31 00:48:14.167663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:14.167519 4519 scope.go:122] "RemoveContainer" containerID="985f3bfd29a7b8f1705cfb0b397b347962bdf17500f388c8ecf5f3fae78123cc" Oct 31 00:48:14.167754 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:14.167737 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs" Oct 31 00:48:14.173219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:14.173187 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" event={"ID":"9e7a3b29-178b-43e4-b127-314c2b183d9e","Type":"ContainerStarted","Data":"363b3e257dbdad72860ea518e1517a2f1958b2d68128f7d2fc68e7944254e04a"} Oct 31 00:48:14.192251 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:14.192208 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs"] Oct 31 00:48:14.199064 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:14.199031 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1d4a7e-649d8bd54c-lq4qs"] Oct 31 00:48:14.271892 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:14.271856 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm"] Oct 31 00:48:14.341435 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:14.341401 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:14.442043 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:14.442002 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:14.501860 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:14.501819 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" podUID="7dd65b23-b72a-4eb4-971f-5f218554d0f2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:48:14.502059 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:14.501885 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" podUID="fd5bf7df-a3e9-4de3-ac84-e258676c6f92" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:48:14.888609 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:14.888580 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:15.181872 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:15.181790 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" event={"ID":"9e7a3b29-178b-43e4-b127-314c2b183d9e","Type":"ContainerStarted","Data":"9e33b8afb51f75baebf9b8dc648d0c80d472fd199f646373c4dfff580adb7b54"} Oct 31 00:48:15.208089 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:15.208035 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" podStartSLOduration=2.208020939 podStartE2EDuration="2.208020939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:15.204748195 +0000 UTC m=+891.233928587" watchObservedRunningTime="2026-10-31 00:48:15.208020939 +0000 UTC m=+891.237201328" Oct 31 00:48:15.699836 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:15.699803 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:15.701345 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:15.701319 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:48:15.701470 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:15.701368 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 00:48:15.750499 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:15.750458 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:16.105357 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:16.105327 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d39d45bb-7e82-4c44-8038-491cf125e145" path="/var/lib/kubelet/pods/d39d45bb-7e82-4c44-8038-491cf125e145/volumes" Oct 31 00:48:16.570804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:16.570759 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:16.703121 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:16.702047 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:48:16.703121 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:16.702125 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 00:48:17.510961 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:17.510770 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:17.706176 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:17.706140 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:48:17.706453 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:17.706423 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 00:48:17.731660 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:48:17.714207 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:48:18.221037 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:18.216941 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" containerID="cri-o://744b19da8f245e6fb4da7b00b6fa9b1b74d6652c41f462af46d954a1999b5df1" gracePeriod=45 Oct 31 00:48:18.434676 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:18.432851 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:18.457659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:18.453727 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:18.465712 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:18.465626 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:18.492668 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:18.488706 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:18.501659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:18.498774 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:18.665661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:18.664748 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:18.691296 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:18.691258 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:18.700776 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:18.700741 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:48:18.701036 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:18.701011 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 00:48:18.956914 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:18.956792 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:19.088797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.088748 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:19.089822 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.089790 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": dial tcp 10.133.0.76:8000: connect: connection refused" Oct 31 00:48:19.090084 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.090065 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:48:19.294770 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.294714 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" podUID="a1a22d4f-183e-4d55-b8f5-71c4d30736bb" containerName="payload-processing" containerID="cri-o://d055e0dd2ce7e62f3e3a0436b35d38ee279a102d2476e730b5f6a5bbe4d56b65" gracePeriod=25 Oct 31 00:48:19.307803 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.307744 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:19.447662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.446719 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:19.459821 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.459769 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:19.486417 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.485845 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:48:19.567654 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.567533 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:48:19.569061 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.569033 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:48:19.577662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.487722 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:19.648725 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.582265 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:48:19.648725 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.583451 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:48:19.648725 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.611862 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:48:19.648725 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.613046 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 00:48:19.648725 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.622186 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:48:19.780850 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.708300 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:19.817372 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.741794 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:48:19.835823 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.817616 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 00:48:19.836194 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:19.836147 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:48:20.035677 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:20.021559 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" containerID="cri-o://467ba0efe527b424c9260b979ab129c36f1f13b277c717465224c584ed0f4cc8" gracePeriod=43 Oct 31 00:48:20.035677 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:20.022031 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" containerID="cri-o://cdd211cf9b890a440e36df4f5b43157b6407fcd6d3929898f39b1efe92400f35" gracePeriod=43 Oct 31 00:48:20.063687 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:20.058402 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" containerID="cri-o://8bd1b63dde235a63d9db6676edf058ab426120cb0d2630ecb3e0b7ac8febc12a" gracePeriod=43 Oct 31 00:48:20.096050 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:20.095883 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 00:48:20.097828 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:20.097800 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:48:20.175084 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:20.175051 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": dial tcp 10.133.0.76:8000: connect: connection refused" Oct 31 00:48:20.540661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:20.540075 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded" Oct 31 00:48:20.565695 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:20.544732 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:20.770661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:20.758186 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:48:20.796726 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:20.773806 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:48:20.796726 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:20.773870 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 00:48:20.796726 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:20.775764 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:48:20.826319 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:20.826272 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:20.861269 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:20.799860 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 00:48:20.862050 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:20.842025 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:48:20.862344 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:20.844270 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:48:21.091549 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.091449 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": dial tcp 10.133.0.76:8000: connect: connection refused" Oct 31 00:48:21.107578 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.105529 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:21.122663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.116155 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:21.255525 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.255490 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:21.274243 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:48:21.259989 4519 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.081s" Oct 31 00:48:21.276924 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.276898 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:48:21.277282 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.277264 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:48:21.277650 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.277617 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:48:21.279529 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.279511 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj"] Oct 31 00:48:21.300393 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.300354 4519 cpu_manager.go:414] "containerMap: removing container" podUID="d39d45bb-7e82-4c44-8038-491cf125e145" containerName="payload-pre-processing" Oct 31 00:48:21.303796 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.303763 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d39d45bb-7e82-4c44-8038-491cf125e145" containerName="payload-pre-processing" Oct 31 00:48:21.304350 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.304323 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d39d45bb-7e82-4c44-8038-491cf125e145" containerName="payload-pre-processing" Oct 31 00:48:21.376935 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.376308 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj"] Oct 31 00:48:21.376935 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.376456 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" Oct 31 00:48:21.425834 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.409356 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:48:21.425834 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.409797 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:21.425834 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.410306 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:21.529714 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.511462 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:21.567358 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.548764 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:21.567358 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.549422 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": dial tcp 10.133.0.67:8000: connect: connection refused" Oct 31 00:48:21.587691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.578953 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:21.598669 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.590393 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece35843-f7a2-481a-b854-8b15bfdbb38f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj\" (UID: \"ece35843-f7a2-481a-b854-8b15bfdbb38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" Oct 31 00:48:21.602664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.599052 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wjlc\" (UniqueName: \"kubernetes.io/projected/ece35843-f7a2-481a-b854-8b15bfdbb38f-kube-api-access-8wjlc\") pod \"maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj\" (UID: \"ece35843-f7a2-481a-b854-8b15bfdbb38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" Oct 31 00:48:21.639710 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.606930 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece35843-f7a2-481a-b854-8b15bfdbb38f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj\" (UID: \"ece35843-f7a2-481a-b854-8b15bfdbb38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" Oct 31 00:48:21.723663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.715790 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:48:21.723663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.715869 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 00:48:21.776672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.768946 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece35843-f7a2-481a-b854-8b15bfdbb38f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj\" (UID: \"ece35843-f7a2-481a-b854-8b15bfdbb38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" Oct 31 00:48:21.793660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.790857 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjlc\" (UniqueName: \"kubernetes.io/projected/ece35843-f7a2-481a-b854-8b15bfdbb38f-kube-api-access-8wjlc\") pod \"maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj\" (UID: \"ece35843-f7a2-481a-b854-8b15bfdbb38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" Oct 31 00:48:21.793660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.791193 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece35843-f7a2-481a-b854-8b15bfdbb38f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj\" (UID: \"ece35843-f7a2-481a-b854-8b15bfdbb38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" Oct 31 00:48:21.793660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.793491 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:48:21.793976 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.793947 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece35843-f7a2-481a-b854-8b15bfdbb38f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj\" (UID: \"ece35843-f7a2-481a-b854-8b15bfdbb38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" Oct 31 00:48:21.794044 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.793367 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:21.807895 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.806620 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:48:21.807895 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.807015 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 31 00:48:21.807895 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.807399 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": dial tcp 10.133.0.45:8000: connect: connection refused" Oct 31 00:48:21.807895 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.807497 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:48:21.871701 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.857776 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": read tcp 10.133.0.2:49900->10.133.0.42:8000: read: connection reset by peer" Oct 31 00:48:21.898767 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.889038 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:48:21.898767 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.889396 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 00:48:21.898767 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.889661 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 00:48:21.898767 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:21.889881 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:48:22.100665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.082562 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wjlc\" (UniqueName: \"kubernetes.io/projected/ece35843-f7a2-481a-b854-8b15bfdbb38f-kube-api-access-8wjlc\") pod \"maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj\" (UID: \"ece35843-f7a2-481a-b854-8b15bfdbb38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" Oct 31 00:48:22.346025 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.102565 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": dial tcp 10.133.0.76:8000: connect: connection refused" Oct 31 00:48:22.346344 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.150742 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 00:48:22.350047 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.206106 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece35843-f7a2-481a-b854-8b15bfdbb38f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj\" (UID: \"ece35843-f7a2-481a-b854-8b15bfdbb38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" Oct 31 00:48:22.457838 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.457790 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": dial tcp 10.133.0.67:8000: connect: connection refused" Oct 31 00:48:22.458352 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.458321 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": dial tcp 10.133.0.45:8000: connect: connection refused" Oct 31 00:48:22.550418 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.550248 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": dial tcp 10.133.0.42:8000: connect: connection refused" Oct 31 00:48:22.587973 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.583854 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6"] Oct 31 00:48:22.587973 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.583920 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6"] Oct 31 00:48:22.587973 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.583947 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p"] Oct 31 00:48:22.587973 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.584143 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" podUID="479718cc-f691-42a3-b6be-8f12366835cc" containerName="istio-proxy" containerID="cri-o://33940ca52a3535c1baf63cfe4dd7870ca997b77a4bc1ae54fd366804a0fda590" gracePeriod=30 Oct 31 00:48:22.587973 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.584589 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" podUID="467bdac4-5015-4338-a6b1-c7591c441847" containerName="istio-proxy" containerID="cri-o://238f611f7c542b7063070f09d96c36047096debb888a7cee60a445f152d69b36" gracePeriod=30 Oct 31 00:48:22.629158 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.600771 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerName="istio-proxy" containerID="cri-o://6f60b4c9e8622704ba2f3440059aadb872b0bab7cfc5020331323d5f4a0d81f6" gracePeriod=30 Oct 31 00:48:22.629158 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.601356 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:48:22.629158 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.602447 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" Oct 31 00:48:22.790825 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.780837 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:48:22.790825 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.780899 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 00:48:22.871820 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:22.813970 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:23.055073 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.055029 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:48:23.056762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.056726 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 00:48:23.064951 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.064912 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 00:48:23.164100 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.118368 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": dial tcp 10.133.0.76:8000: connect: connection refused" Oct 31 00:48:23.378246 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.378206 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:23.430982 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.430894 4519 patch_prober.go:29] interesting pod/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:48:23.431228 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.431201 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" podUID="467bdac4-5015-4338-a6b1-c7591c441847" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:23.433359 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.433326 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:23.434554 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.434525 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": dial tcp 10.133.0.67:8000: connect: connection refused" Oct 31 00:48:23.435096 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.435070 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": dial tcp 10.133.0.45:8000: connect: connection refused" Oct 31 00:48:23.483885 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.483845 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:48:23.484167 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.484136 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" podUID="cdbf7b30-65d4-4462-af10-05cdd5015ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:23.528726 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.497979 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": dial tcp 10.133.0.42:8000: connect: connection refused" Oct 31 00:48:23.696059 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.695971 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:48:23.696328 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.696276 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" podUID="cbd4f0b9-e235-40f7-acef-68e1051fc918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:23.729921 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.729883 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:48:23.730168 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.730142 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 00:48:23.818956 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.818892 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:23.909662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.876766 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:23.993656 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:23.990560 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:24.085661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:24.084722 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:24.085661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:24.085431 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:48:24.123785 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:24.107795 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" containerID="cri-o://7a23d0c2b116f2e5ef532751a0b422ea505f03a538521e0f512081246a29cc6a" gracePeriod=45 Oct 31 00:48:24.123785 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:24.108595 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 00:48:24.167699 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:24.153287 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": dial tcp 10.133.0.76:8000: connect: connection refused" Oct 31 00:48:24.182692 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:24.168439 4519 generic.go:356] "Generic (PLEG): container finished" podID="2f863a10-a9a8-4894-ac86-2f057371b472" containerID="744b19da8f245e6fb4da7b00b6fa9b1b74d6652c41f462af46d954a1999b5df1" exitCode=0 Oct 31 00:48:24.307814 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:24.307762 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" event={"ID":"2f863a10-a9a8-4894-ac86-2f057371b472","Type":"ContainerDied","Data":"744b19da8f245e6fb4da7b00b6fa9b1b74d6652c41f462af46d954a1999b5df1"} Oct 31 00:48:24.308245 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:24.308206 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb"] Oct 31 00:48:24.513690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:24.486935 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 00:48:24.573201 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:24.525080 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:48:24.573201 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:24.525138 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" podUID="f6b39483-2371-4570-843b-c4421cad360f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:24.653592 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:24.653553 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": read tcp 10.133.0.2:46370->10.133.0.40:8000: read: connection reset by peer" Oct 31 00:48:24.821192 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:24.738038 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": dial tcp 10.133.0.40:8000: connect: connection refused" Oct 31 00:48:24.821192 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:24.778455 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:48:24.821192 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:24.778509 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 00:48:24.821192 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:24.806962 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": dial tcp 10.133.0.42:8000: connect: connection refused" Oct 31 00:48:25.103848 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.103737 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:25.144899 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.140294 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": dial tcp 10.133.0.67:8000: connect: connection refused" Oct 31 00:48:25.144899 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.141774 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:48:25.145758 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.145725 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:25.146188 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.146166 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:48:25.221330 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.219732 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" podUID="7dd65b23-b72a-4eb4-971f-5f218554d0f2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:48:25.267539 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.263554 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj"] Oct 31 00:48:25.267539 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.266886 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": dial tcp 10.133.0.45:8000: connect: connection refused" Oct 31 00:48:25.309434 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.309295 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:48:25.350415 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.350379 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:25.350965 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.350934 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" podUID="a1a22d4f-183e-4d55-b8f5-71c4d30736bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:48:25.375509 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.374252 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": dial tcp 10.133.0.76:8000: connect: connection refused" Oct 31 00:48:25.412659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.406245 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 00:48:25.443285 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.435239 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" podUID="fd5bf7df-a3e9-4de3-ac84-e258676c6f92" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:48:25.489120 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.489061 4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:25.506829 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.505956 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": dial tcp 10.133.0.45:8000: connect: connection refused" Oct 31 00:48:25.506829 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.506073 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": dial tcp 10.133.0.67:8000: connect: connection refused" Oct 31 00:48:25.539416 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.530913 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:48:25.539745 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.530974 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" podUID="4edab197-6a3f-457a-acab-82d4cebc7b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:25.631254 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.630106 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:48:25.631526 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.631509 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:48:25.646889 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.646848 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": dial tcp 10.133.0.42:8000: connect: connection refused" Oct 31 00:48:25.653353 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.653314 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:25.661813 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.661776 4519 generic.go:356] "Generic (PLEG): container finished" podID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerID="cdd211cf9b890a440e36df4f5b43157b6407fcd6d3929898f39b1efe92400f35" exitCode=0 Oct 31 00:48:25.662107 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.662085 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" event={"ID":"b2a6c2be-3886-4f44-a65a-7eef406c0a45","Type":"ContainerDied","Data":"cdd211cf9b890a440e36df4f5b43157b6407fcd6d3929898f39b1efe92400f35"} Oct 31 00:48:25.664813 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.664782 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": dial tcp 10.133.0.40:8000: connect: connection refused" Oct 31 00:48:25.680883 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.680850 4519 generic.go:356] "Generic (PLEG): container finished" podID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerID="8bd1b63dde235a63d9db6676edf058ab426120cb0d2630ecb3e0b7ac8febc12a" exitCode=0 Oct 31 00:48:25.684947 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.682120 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" event={"ID":"886c14ce-3e7b-491b-af66-4a7c9b119861","Type":"ContainerDied","Data":"8bd1b63dde235a63d9db6676edf058ab426120cb0d2630ecb3e0b7ac8febc12a"} Oct 31 00:48:25.708337 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.708308 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:48:25.708555 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.708532 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 00:48:25.782097 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:25.781273 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:48:26.307140 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.306187 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw"] Oct 31 00:48:26.341183 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.340517 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" Oct 31 00:48:26.355456 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.355415 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw"] Oct 31 00:48:26.404518 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.389646 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:48:26.435792 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.435529 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": dial tcp 10.133.0.67:8000: connect: connection refused" Oct 31 00:48:26.487311 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.483839 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfh6d\" (UniqueName: \"kubernetes.io/projected/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-kube-api-access-zfh6d\") pod \"maas-api-revoke-sub-41dbd2ef-gdxqw\" (UID: \"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" Oct 31 00:48:26.487311 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.483986 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41dbd2ef-gdxqw\" (UID: \"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" Oct 31 00:48:26.487311 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.484037 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41dbd2ef-gdxqw\" (UID: \"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" Oct 31 00:48:26.572969 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.572835 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:26.583331 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.583309 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:48:26.585055 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.585035 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f863a10-a9a8-4894-ac86-2f057371b472-kube-api-access-t9npw\" (UniqueName: \"kubernetes.io/projected/2f863a10-a9a8-4894-ac86-2f057371b472-kube-api-access-t9npw\") pod \"2f863a10-a9a8-4894-ac86-2f057371b472\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " Oct 31 00:48:26.585482 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.585289 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f863a10-a9a8-4894-ac86-2f057371b472-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f863a10-a9a8-4894-ac86-2f057371b472-tls-certs\") pod \"2f863a10-a9a8-4894-ac86-2f057371b472\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " Oct 31 00:48:26.585782 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.585755 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-tmp-dir\") pod \"2f863a10-a9a8-4894-ac86-2f057371b472\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " Oct 31 00:48:26.585889 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.585880 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-home\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-home\") pod \"2f863a10-a9a8-4894-ac86-2f057371b472\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " Oct 31 00:48:26.586026 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.586006 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-model-cache\") pod \"2f863a10-a9a8-4894-ac86-2f057371b472\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " Oct 31 00:48:26.586134 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.586125 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-dshm\") pod \"2f863a10-a9a8-4894-ac86-2f057371b472\" (UID: \"2f863a10-a9a8-4894-ac86-2f057371b472\") " Oct 31 00:48:26.586600 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.586579 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfh6d\" (UniqueName: \"kubernetes.io/projected/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-kube-api-access-zfh6d\") pod \"maas-api-revoke-sub-41dbd2ef-gdxqw\" (UID: \"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" Oct 31 00:48:26.587293 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.587261 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41dbd2ef-gdxqw\" (UID: \"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" Oct 31 00:48:26.587476 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.587447 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41dbd2ef-gdxqw\" (UID: \"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" Oct 31 00:48:26.593770 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.587336 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-tmp-dir" pod "2f863a10-a9a8-4894-ac86-2f057371b472" (UID: "2f863a10-a9a8-4894-ac86-2f057371b472"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:26.593993 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.588609 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41dbd2ef-gdxqw\" (UID: \"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" Oct 31 00:48:26.593993 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.588580 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-home" pod "2f863a10-a9a8-4894-ac86-2f057371b472" (UID: "2f863a10-a9a8-4894-ac86-2f057371b472"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:26.593993 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.588822 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-model-cache" pod "2f863a10-a9a8-4894-ac86-2f057371b472" (UID: "2f863a10-a9a8-4894-ac86-2f057371b472"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:26.593993 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.593240 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f863a10-a9a8-4894-ac86-2f057371b472-tls-certs" pod "2f863a10-a9a8-4894-ac86-2f057371b472" (UID: "2f863a10-a9a8-4894-ac86-2f057371b472"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:26.598839 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.598750 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:48:26.600262 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.600237 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfh6d\" (UniqueName: \"kubernetes.io/projected/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-kube-api-access-zfh6d\") pod \"maas-api-revoke-sub-41dbd2ef-gdxqw\" (UID: \"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" Oct 31 00:48:26.600574 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.600550 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f863a10-a9a8-4894-ac86-2f057371b472-kube-api-access-t9npw" pod "2f863a10-a9a8-4894-ac86-2f057371b472" (UID: "2f863a10-a9a8-4894-ac86-2f057371b472"). InnerVolumeSpecName "kube-api-access-t9npw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:26.601313 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.601286 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-dshm" pod "2f863a10-a9a8-4894-ac86-2f057371b472" (UID: "2f863a10-a9a8-4894-ac86-2f057371b472"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:26.603606 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.603578 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41dbd2ef-gdxqw\" (UID: \"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.697883 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-home\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-home\") pod \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.697980 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2a6c2be-3886-4f44-a65a-7eef406c0a45-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a6c2be-3886-4f44-a65a-7eef406c0a45-tls-certs\") pod \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.698033 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-home\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-home\") pod \"886c14ce-3e7b-491b-af66-4a7c9b119861\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.698061 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-tmp-dir\") pod \"886c14ce-3e7b-491b-af66-4a7c9b119861\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.698206 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-tmp-dir\") pod \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.698238 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/886c14ce-3e7b-491b-af66-4a7c9b119861-kube-api-access-g66sl\" (UniqueName: \"kubernetes.io/projected/886c14ce-3e7b-491b-af66-4a7c9b119861-kube-api-access-g66sl\") pod \"886c14ce-3e7b-491b-af66-4a7c9b119861\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.698270 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/886c14ce-3e7b-491b-af66-4a7c9b119861-tls-certs\" (UniqueName: \"kubernetes.io/secret/886c14ce-3e7b-491b-af66-4a7c9b119861-tls-certs\") pod \"886c14ce-3e7b-491b-af66-4a7c9b119861\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.698305 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-model-cache\") pod \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.698426 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a6c2be-3886-4f44-a65a-7eef406c0a45-kube-api-access-xz9x4\" (UniqueName: \"kubernetes.io/projected/b2a6c2be-3886-4f44-a65a-7eef406c0a45-kube-api-access-xz9x4\") pod \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.698462 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-model-cache\") pod \"886c14ce-3e7b-491b-af66-4a7c9b119861\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.698505 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-dshm\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-dshm\") pod \"886c14ce-3e7b-491b-af66-4a7c9b119861\" (UID: \"886c14ce-3e7b-491b-af66-4a7c9b119861\") " Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.698586 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-dshm\") pod \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\" (UID: \"b2a6c2be-3886-4f44-a65a-7eef406c0a45\") " Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.739739 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.739799 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.745537 4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-model-cache\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.745572 4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-dshm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.745588 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9npw\" (UniqueName: \"kubernetes.io/projected/2f863a10-a9a8-4894-ac86-2f057371b472-kube-api-access-t9npw\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.745604 4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f863a10-a9a8-4894-ac86-2f057371b472-tls-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.745618 4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-tmp-dir\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.745727 4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f863a10-a9a8-4894-ac86-2f057371b472-home\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.746051 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-home" pod "886c14ce-3e7b-491b-af66-4a7c9b119861" (UID: "886c14ce-3e7b-491b-af66-4a7c9b119861"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.746270 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-home" pod "b2a6c2be-3886-4f44-a65a-7eef406c0a45" (UID: "b2a6c2be-3886-4f44-a65a-7eef406c0a45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.758188 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a6c2be-3886-4f44-a65a-7eef406c0a45-tls-certs" pod "b2a6c2be-3886-4f44-a65a-7eef406c0a45" (UID: "b2a6c2be-3886-4f44-a65a-7eef406c0a45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:26.767673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.767630 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-tmp-dir" pod "886c14ce-3e7b-491b-af66-4a7c9b119861" (UID: "886c14ce-3e7b-491b-af66-4a7c9b119861"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:26.769126 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.767865 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-model-cache" pod "886c14ce-3e7b-491b-af66-4a7c9b119861" (UID: "886c14ce-3e7b-491b-af66-4a7c9b119861"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:26.769126 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.768043 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-model-cache" pod "b2a6c2be-3886-4f44-a65a-7eef406c0a45" (UID: "b2a6c2be-3886-4f44-a65a-7eef406c0a45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:26.770038 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.769778 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" Oct 31 00:48:26.852306 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.791864 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-tmp-dir" pod "b2a6c2be-3886-4f44-a65a-7eef406c0a45" (UID: "b2a6c2be-3886-4f44-a65a-7eef406c0a45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:26.861403 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.858699 4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-home\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.861403 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.858729 4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a6c2be-3886-4f44-a65a-7eef406c0a45-tls-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.861403 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.858744 4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-home\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.861403 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.858761 4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-tmp-dir\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.861403 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.858776 4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-tmp-dir\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.861403 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.858789 4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-model-cache\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.861403 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.858810 4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-model-cache\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.864651 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.864602 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" Oct 31 00:48:26.871622 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.871587 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" event={"ID":"2f863a10-a9a8-4894-ac86-2f057371b472","Type":"ContainerDied","Data":"87b8111251cc88a0f023db124c84a0addd5a63e1d1ab542e8445bcba8442af36"} Oct 31 00:48:26.871777 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.871660 4519 scope.go:122] "RemoveContainer" containerID="744b19da8f245e6fb4da7b00b6fa9b1b74d6652c41f462af46d954a1999b5df1" Oct 31 00:48:26.875335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.875180 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-dshm" pod "b2a6c2be-3886-4f44-a65a-7eef406c0a45" (UID: "b2a6c2be-3886-4f44-a65a-7eef406c0a45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:26.876368 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.876022 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/886c14ce-3e7b-491b-af66-4a7c9b119861-tls-certs" pod "886c14ce-3e7b-491b-af66-4a7c9b119861" (UID: "886c14ce-3e7b-491b-af66-4a7c9b119861"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:26.876368 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.876231 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886c14ce-3e7b-491b-af66-4a7c9b119861-kube-api-access-g66sl" pod "886c14ce-3e7b-491b-af66-4a7c9b119861" (UID: "886c14ce-3e7b-491b-af66-4a7c9b119861"). InnerVolumeSpecName "kube-api-access-g66sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:26.880804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.880780 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a6c2be-3886-4f44-a65a-7eef406c0a45-kube-api-access-xz9x4" pod "b2a6c2be-3886-4f44-a65a-7eef406c0a45" (UID: "b2a6c2be-3886-4f44-a65a-7eef406c0a45"). InnerVolumeSpecName "kube-api-access-xz9x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:26.886195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.881988 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" event={"ID":"b2a6c2be-3886-4f44-a65a-7eef406c0a45","Type":"ContainerDied","Data":"2d8bcd0825778a5010bddf6d5280548f04105d67e8148fc650aff1489fdb1de3"} Oct 31 00:48:26.886195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.882099 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" Oct 31 00:48:26.886195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.886012 4519 generic.go:356] "Generic (PLEG): container finished" podID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerID="7a23d0c2b116f2e5ef532751a0b422ea505f03a538521e0f512081246a29cc6a" exitCode=0 Oct 31 00:48:26.886195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.886093 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" event={"ID":"804eee3f-86ba-4511-a9c2-279edb91cf38","Type":"ContainerDied","Data":"7a23d0c2b116f2e5ef532751a0b422ea505f03a538521e0f512081246a29cc6a"} Oct 31 00:48:26.886589 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.886565 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-dshm" pod "886c14ce-3e7b-491b-af66-4a7c9b119861" (UID: "886c14ce-3e7b-491b-af66-4a7c9b119861"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:26.888582 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.888556 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" event={"ID":"ece35843-f7a2-481a-b854-8b15bfdbb38f","Type":"ContainerStarted","Data":"fb53b0a9b9810d8a7ffd6a203d3afac072db3616f9ba39c2528821e3c2e0d301"} Oct 31 00:48:26.888699 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.888612 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" event={"ID":"ece35843-f7a2-481a-b854-8b15bfdbb38f","Type":"ContainerStarted","Data":"2e71fde1c0a3cbd9263b816fb4a97080e0e9efa5f9f2e0a6dac03d9b20abd719"} Oct 31 00:48:26.897693 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.897548 4519 generic.go:356] "Generic (PLEG): container finished" podID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerID="467ba0efe527b424c9260b979ab129c36f1f13b277c717465224c584ed0f4cc8" exitCode=0 Oct 31 00:48:26.897693 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.897680 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" event={"ID":"afa77fbd-2015-4986-bf98-f4858d87f0bf","Type":"ContainerDied","Data":"467ba0efe527b424c9260b979ab129c36f1f13b277c717465224c584ed0f4cc8"} Oct 31 00:48:26.901028 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.900126 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" event={"ID":"886c14ce-3e7b-491b-af66-4a7c9b119861","Type":"ContainerDied","Data":"22f86af2ed1e2dce39795a107c2be459e6a9ff4550e52e5aff976d6c8a311ee3"} Oct 31 00:48:26.901028 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.900212 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" Oct 31 00:48:26.918077 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.918012 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" podStartSLOduration=7.91799247 podStartE2EDuration="7.91799247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:26.915444873 +0000 UTC m=+902.944625264" watchObservedRunningTime="2026-10-31 00:48:26.91799247 +0000 UTC m=+902.947172862" Oct 31 00:48:26.942247 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.942177 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:48:26.942434 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.942415 4519 scope.go:122] "RemoveContainer" containerID="cdd211cf9b890a440e36df4f5b43157b6407fcd6d3929898f39b1efe92400f35" Oct 31 00:48:26.955528 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.955500 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:48:26.961503 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.961476 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g66sl\" (UniqueName: \"kubernetes.io/projected/886c14ce-3e7b-491b-af66-4a7c9b119861-kube-api-access-g66sl\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.961617 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.961510 4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/886c14ce-3e7b-491b-af66-4a7c9b119861-tls-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.961617 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.961525 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz9x4\" (UniqueName: \"kubernetes.io/projected/b2a6c2be-3886-4f44-a65a-7eef406c0a45-kube-api-access-xz9x4\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.961617 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.961540 4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/886c14ce-3e7b-491b-af66-4a7c9b119861-dshm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.961617 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.961552 4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a6c2be-3886-4f44-a65a-7eef406c0a45-dshm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.977827 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.977750 4519 scope.go:122] "RemoveContainer" containerID="8bd1b63dde235a63d9db6676edf058ab426120cb0d2630ecb3e0b7ac8febc12a" Oct 31 00:48:26.981201 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.981177 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx"] Oct 31 00:48:26.987698 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.987083 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx"] Oct 31 00:48:26.995181 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:26.995146 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj"] Oct 31 00:48:27.001523 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.001490 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj"] Oct 31 00:48:27.061949 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.061913 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc"] Oct 31 00:48:27.062430 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.062404 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-tmp-dir\") pod \"804eee3f-86ba-4511-a9c2-279edb91cf38\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " Oct 31 00:48:27.062493 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.062448 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-home\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-home\") pod \"804eee3f-86ba-4511-a9c2-279edb91cf38\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " Oct 31 00:48:27.062548 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.062514 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-model-cache\") pod \"804eee3f-86ba-4511-a9c2-279edb91cf38\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " Oct 31 00:48:27.062615 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.062560 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-dshm\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-dshm\") pod \"804eee3f-86ba-4511-a9c2-279edb91cf38\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " Oct 31 00:48:27.062721 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.062682 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afa77fbd-2015-4986-bf98-f4858d87f0bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/afa77fbd-2015-4986-bf98-f4858d87f0bf-tls-certs\") pod \"afa77fbd-2015-4986-bf98-f4858d87f0bf\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " Oct 31 00:48:27.062785 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.062737 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-model-cache\") pod \"afa77fbd-2015-4986-bf98-f4858d87f0bf\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " Oct 31 00:48:27.062882 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.062864 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804eee3f-86ba-4511-a9c2-279edb91cf38-kube-api-access-zt9d9\" (UniqueName: \"kubernetes.io/projected/804eee3f-86ba-4511-a9c2-279edb91cf38-kube-api-access-zt9d9\") pod \"804eee3f-86ba-4511-a9c2-279edb91cf38\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " Oct 31 00:48:27.062937 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.062887 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-model-cache" pod "804eee3f-86ba-4511-a9c2-279edb91cf38" (UID: "804eee3f-86ba-4511-a9c2-279edb91cf38"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:27.062937 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.062905 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-tmp-dir\") pod \"afa77fbd-2015-4986-bf98-f4858d87f0bf\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " Oct 31 00:48:27.063043 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.062972 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa77fbd-2015-4986-bf98-f4858d87f0bf-kube-api-access-ndptw\" (UniqueName: \"kubernetes.io/projected/afa77fbd-2015-4986-bf98-f4858d87f0bf-kube-api-access-ndptw\") pod \"afa77fbd-2015-4986-bf98-f4858d87f0bf\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " Oct 31 00:48:27.063043 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.063029 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-home\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-home\") pod \"afa77fbd-2015-4986-bf98-f4858d87f0bf\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " Oct 31 00:48:27.063187 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.063067 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-dshm\") pod \"afa77fbd-2015-4986-bf98-f4858d87f0bf\" (UID: \"afa77fbd-2015-4986-bf98-f4858d87f0bf\") " Oct 31 00:48:27.063187 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.063088 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-tmp-dir" pod "804eee3f-86ba-4511-a9c2-279edb91cf38" (UID: "804eee3f-86ba-4511-a9c2-279edb91cf38"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:27.063187 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.063097 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/804eee3f-86ba-4511-a9c2-279edb91cf38-tls-certs\" (UniqueName: \"kubernetes.io/secret/804eee3f-86ba-4511-a9c2-279edb91cf38-tls-certs\") pod \"804eee3f-86ba-4511-a9c2-279edb91cf38\" (UID: \"804eee3f-86ba-4511-a9c2-279edb91cf38\") " Oct 31 00:48:27.063330 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.063226 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-home" pod "804eee3f-86ba-4511-a9c2-279edb91cf38" (UID: "804eee3f-86ba-4511-a9c2-279edb91cf38"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:27.063971 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.063947 4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-tmp-dir\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:27.064146 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.064129 4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-home\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:27.064244 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.064231 4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-model-cache\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:27.064350 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.063974 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-model-cache" pod "afa77fbd-2015-4986-bf98-f4858d87f0bf" (UID: "afa77fbd-2015-4986-bf98-f4858d87f0bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:27.064449 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.064181 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-tmp-dir" pod "afa77fbd-2015-4986-bf98-f4858d87f0bf" (UID: "afa77fbd-2015-4986-bf98-f4858d87f0bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:27.064890 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.064827 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-home" pod "afa77fbd-2015-4986-bf98-f4858d87f0bf" (UID: "afa77fbd-2015-4986-bf98-f4858d87f0bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:27.066965 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.066943 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804eee3f-86ba-4511-a9c2-279edb91cf38-kube-api-access-zt9d9" pod "804eee3f-86ba-4511-a9c2-279edb91cf38" (UID: "804eee3f-86ba-4511-a9c2-279edb91cf38"). InnerVolumeSpecName "kube-api-access-zt9d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:27.067231 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.067137 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa77fbd-2015-4986-bf98-f4858d87f0bf-tls-certs" pod "afa77fbd-2015-4986-bf98-f4858d87f0bf" (UID: "afa77fbd-2015-4986-bf98-f4858d87f0bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:27.067231 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.067201 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/804eee3f-86ba-4511-a9c2-279edb91cf38-tls-certs" pod "804eee3f-86ba-4511-a9c2-279edb91cf38" (UID: "804eee3f-86ba-4511-a9c2-279edb91cf38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:27.067942 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.067912 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa77fbd-2015-4986-bf98-f4858d87f0bf-kube-api-access-ndptw" pod "afa77fbd-2015-4986-bf98-f4858d87f0bf" (UID: "afa77fbd-2015-4986-bf98-f4858d87f0bf"). InnerVolumeSpecName "kube-api-access-ndptw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:27.068675 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.068605 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-dshm" pod "afa77fbd-2015-4986-bf98-f4858d87f0bf" (UID: "afa77fbd-2015-4986-bf98-f4858d87f0bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:27.068675 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.068664 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-dshm" pod "804eee3f-86ba-4511-a9c2-279edb91cf38" (UID: "804eee3f-86ba-4511-a9c2-279edb91cf38"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:27.073449 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.073424 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc"] Oct 31 00:48:27.089332 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.089290 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/distinct-2-w5-44bc67-kserve-85d77ffc4c-6bzfx" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 00:48:27.142786 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.142692 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:27.165389 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.165347 4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/804eee3f-86ba-4511-a9c2-279edb91cf38-dshm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:27.165389 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.165381 4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa77fbd-2015-4986-bf98-f4858d87f0bf-tls-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:27.165389 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.165395 4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-model-cache\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:27.165630 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.165410 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt9d9\" (UniqueName: \"kubernetes.io/projected/804eee3f-86ba-4511-a9c2-279edb91cf38-kube-api-access-zt9d9\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:27.165630 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.165422 4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-tmp-dir\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:27.165630 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.165434 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndptw\" (UniqueName: \"kubernetes.io/projected/afa77fbd-2015-4986-bf98-f4858d87f0bf-kube-api-access-ndptw\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:27.165630 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.165449 4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-home\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:27.165630 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.165461 4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa77fbd-2015-4986-bf98-f4858d87f0bf-dshm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:27.165630 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.165472 4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/804eee3f-86ba-4511-a9c2-279edb91cf38-tls-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:27.195486 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:48:27.194407 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb01356_7f9b_4d24_9f36_f4ad1fa75e5f.slice/crio-98eb211e0ec0b5f6e0bfc8f1a6c498f1f65009b8b0fff49a7014f5b4adf19441 WatchSource:0}: Error finding container 98eb211e0ec0b5f6e0bfc8f1a6c498f1f65009b8b0fff49a7014f5b4adf19441: Status 404 returned error can't find the container with id 98eb211e0ec0b5f6e0bfc8f1a6c498f1f65009b8b0fff49a7014f5b4adf19441 Oct 31 00:48:27.207050 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.206990 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw"] Oct 31 00:48:27.327940 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.327906 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:27.398775 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.398680 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd"] Oct 31 00:48:27.402338 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.402309 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-1d4a7e-vsckd"] Oct 31 00:48:27.427966 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.427496 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:27.434485 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.434448 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/premium-w5-44bc67-kserve-599494fcf7-hgztj" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:27.497153 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.497104 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-44bc67/simulator-w5-44bc67-kserve-6c77894df6-5kbvc" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:48:27.522173 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.522132 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:27.635674 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.635363 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:27.674825 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.674612 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 00:48:27.702075 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.702037 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:48:27.702316 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.702290 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 00:48:27.802288 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.802203 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:27.999330 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:27.998939 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:28.006766 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.006733 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" event={"ID":"804eee3f-86ba-4511-a9c2-279edb91cf38","Type":"ContainerDied","Data":"402dbebe6d0843f29deacf94c0a6b9fe3342a486b6d22a570968cbc4cf406ffc"} Oct 31 00:48:28.007277 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.006936 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d" Oct 31 00:48:28.007931 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.007911 4519 scope.go:122] "RemoveContainer" containerID="7a23d0c2b116f2e5ef532751a0b422ea505f03a538521e0f512081246a29cc6a" Oct 31 00:48:28.011387 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.011356 4519 generic.go:356] "Generic (PLEG): container finished" podID="ece35843-f7a2-481a-b854-8b15bfdbb38f" containerID="fb53b0a9b9810d8a7ffd6a203d3afac072db3616f9ba39c2528821e3c2e0d301" exitCode=0 Oct 31 00:48:28.011592 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.011567 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" event={"ID":"ece35843-f7a2-481a-b854-8b15bfdbb38f","Type":"ContainerDied","Data":"fb53b0a9b9810d8a7ffd6a203d3afac072db3616f9ba39c2528821e3c2e0d301"} Oct 31 00:48:28.016467 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.016443 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" event={"ID":"afa77fbd-2015-4986-bf98-f4858d87f0bf","Type":"ContainerDied","Data":"d39da3c482e8576f1b75bcaba42ffe7b4a769c332c36558e04fd060b0f8db62b"} Oct 31 00:48:28.016943 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.016712 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp" Oct 31 00:48:28.037615 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.037576 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" event={"ID":"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f","Type":"ContainerStarted","Data":"4c3a3dc9dca290beeaaf3e31c75798d9b3951c6ff8c37e3964934cc8ed5cc71a"} Oct 31 00:48:28.037804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.037683 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" event={"ID":"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f","Type":"ContainerStarted","Data":"98eb211e0ec0b5f6e0bfc8f1a6c498f1f65009b8b0fff49a7014f5b4adf19441"} Oct 31 00:48:28.047472 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.047441 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln"] Oct 31 00:48:28.058793 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.058732 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-44bc67-pj7ln"] Oct 31 00:48:28.086106 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.086036 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:28.208661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.203815 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" path="/var/lib/kubelet/pods/2f863a10-a9a8-4894-ac86-2f057371b472/volumes" Oct 31 00:48:28.208661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.204536 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" path="/var/lib/kubelet/pods/886c14ce-3e7b-491b-af66-4a7c9b119861/volumes" Oct 31 00:48:28.208661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.205147 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d7a5f83-6b68-484a-a30f-9f59eca82c4e" path="/var/lib/kubelet/pods/8d7a5f83-6b68-484a-a30f-9f59eca82c4e/volumes" Oct 31 00:48:28.208661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.206217 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9" path="/var/lib/kubelet/pods/a99cc99c-4c6c-45b8-b96a-a45d78e9c5c9/volumes" Oct 31 00:48:28.208661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.206838 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" path="/var/lib/kubelet/pods/b2a6c2be-3886-4f44-a65a-7eef406c0a45/volumes" Oct 31 00:48:28.215763 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.215513 4519 scope.go:122] "RemoveContainer" containerID="467ba0efe527b424c9260b979ab129c36f1f13b277c717465224c584ed0f4cc8" Oct 31 00:48:28.232599 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.232491 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" podStartSLOduration=2.232472025 podStartE2EDuration="2.232472025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:28.084774629 +0000 UTC m=+904.113955021" watchObservedRunningTime="2026-10-31 00:48:28.232472025 +0000 UTC m=+904.261652416" Oct 31 00:48:28.233613 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.233585 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp"] Oct 31 00:48:28.239978 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.239956 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-44bc67/distinct-w5-44bc67-kserve-59979f5cd8-knbmp"] Oct 31 00:48:28.268260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.268227 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d"] Oct 31 00:48:28.273041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.273016 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-7f1496/simulator-w1-7f1496-kserve-6b4cf66bb9-5dn2d"] Oct 31 00:48:28.476284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.476249 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:28.542479 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.542454 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:48:28.546854 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.546828 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:48:28.551232 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.551212 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:48:28.624703 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.624630 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/479718cc-f691-42a3-b6be-8f12366835cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479718cc-f691-42a3-b6be-8f12366835cc-istio-podinfo\") pod \"479718cc-f691-42a3-b6be-8f12366835cc\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " Oct 31 00:48:28.624957 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.624935 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-data\") pod \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " Oct 31 00:48:28.625127 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.625110 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-workload-certs\") pod \"479718cc-f691-42a3-b6be-8f12366835cc\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " Oct 31 00:48:28.625298 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.625261 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-istio-data\") pod \"467bdac4-5015-4338-a6b1-c7591c441847\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " Oct 31 00:48:28.625432 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.625417 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istiod-ca-cert\") pod \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " Oct 31 00:48:28.625552 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.625536 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-credential-socket\") pod \"479718cc-f691-42a3-b6be-8f12366835cc\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " Oct 31 00:48:28.626193 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.626168 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-podinfo\") pod \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " Oct 31 00:48:28.626380 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.626362 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-workload-socket\") pod \"479718cc-f691-42a3-b6be-8f12366835cc\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " Oct 31 00:48:28.626495 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.626480 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-credential-socket\") pod \"467bdac4-5015-4338-a6b1-c7591c441847\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " Oct 31 00:48:28.626857 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.626836 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-workload-certs\") pod \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " Oct 31 00:48:28.627022 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.627004 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-workload-socket\") pod \"467bdac4-5015-4338-a6b1-c7591c441847\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " Oct 31 00:48:28.627191 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.627174 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/467bdac4-5015-4338-a6b1-c7591c441847-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467bdac4-5015-4338-a6b1-c7591c441847-istio-podinfo\") pod \"467bdac4-5015-4338-a6b1-c7591c441847\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " Oct 31 00:48:28.627322 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.627307 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-kube-api-access-kk6zq\" (UniqueName: \"kubernetes.io/projected/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-kube-api-access-kk6zq\") pod \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " Oct 31 00:48:28.627434 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.627419 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479718cc-f691-42a3-b6be-8f12366835cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479718cc-f691-42a3-b6be-8f12366835cc-istiod-ca-cert\") pod \"479718cc-f691-42a3-b6be-8f12366835cc\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " Oct 31 00:48:28.627714 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.627319 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-workload-socket" pod "467bdac4-5015-4338-a6b1-c7591c441847" (UID: "467bdac4-5015-4338-a6b1-c7591c441847"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:28.627866 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.627484 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-workload-certs" pod "a3eab170-3ff7-48a5-8f32-9acfcb6db28b" (UID: "a3eab170-3ff7-48a5-8f32-9acfcb6db28b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:28.627866 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.627528 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-istio-envoy\") pod \"479718cc-f691-42a3-b6be-8f12366835cc\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " Oct 31 00:48:28.628008 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.627927 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479718cc-f691-42a3-b6be-8f12366835cc-kube-api-access-ffgwf\" (UniqueName: \"kubernetes.io/projected/479718cc-f691-42a3-b6be-8f12366835cc-kube-api-access-ffgwf\") pod \"479718cc-f691-42a3-b6be-8f12366835cc\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " Oct 31 00:48:28.628008 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.627968 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467bdac4-5015-4338-a6b1-c7591c441847-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467bdac4-5015-4338-a6b1-c7591c441847-istiod-ca-cert\") pod \"467bdac4-5015-4338-a6b1-c7591c441847\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " Oct 31 00:48:28.628115 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.628008 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467bdac4-5015-4338-a6b1-c7591c441847-kube-api-access-l947c\" (UniqueName: \"kubernetes.io/projected/467bdac4-5015-4338-a6b1-c7591c441847-kube-api-access-l947c\") pod \"467bdac4-5015-4338-a6b1-c7591c441847\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " Oct 31 00:48:28.628115 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.628042 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-workload-certs\") pod \"467bdac4-5015-4338-a6b1-c7591c441847\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " Oct 31 00:48:28.628115 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.628077 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-credential-socket\") pod \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " Oct 31 00:48:28.628287 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.628151 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479718cc-f691-42a3-b6be-8f12366835cc-istio-token\" (UniqueName: \"kubernetes.io/projected/479718cc-f691-42a3-b6be-8f12366835cc-istio-token\") pod \"479718cc-f691-42a3-b6be-8f12366835cc\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " Oct 31 00:48:28.628287 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.628220 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-istio-data\") pod \"479718cc-f691-42a3-b6be-8f12366835cc\" (UID: \"479718cc-f691-42a3-b6be-8f12366835cc\") " Oct 31 00:48:28.628287 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.628249 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-token\" (UniqueName: \"kubernetes.io/projected/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-token\") pod \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " Oct 31 00:48:28.628287 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.628277 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-workload-socket\") pod \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " Oct 31 00:48:28.631560 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.628340 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-istio-envoy\") pod \"467bdac4-5015-4338-a6b1-c7591c441847\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " Oct 31 00:48:28.631560 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.628376 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467bdac4-5015-4338-a6b1-c7591c441847-istio-token\" (UniqueName: \"kubernetes.io/projected/467bdac4-5015-4338-a6b1-c7591c441847-istio-token\") pod \"467bdac4-5015-4338-a6b1-c7591c441847\" (UID: \"467bdac4-5015-4338-a6b1-c7591c441847\") " Oct 31 00:48:28.631560 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.628461 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-envoy\") pod \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\" (UID: \"a3eab170-3ff7-48a5-8f32-9acfcb6db28b\") " Oct 31 00:48:28.631560 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.629532 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.631560 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.629559 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.632191 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.631868 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-podinfo" pod "a3eab170-3ff7-48a5-8f32-9acfcb6db28b" (UID: "a3eab170-3ff7-48a5-8f32-9acfcb6db28b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:48:28.632506 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.632476 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-workload-socket" pod "a3eab170-3ff7-48a5-8f32-9acfcb6db28b" (UID: "a3eab170-3ff7-48a5-8f32-9acfcb6db28b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:28.632762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.632736 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-istio-data" pod "479718cc-f691-42a3-b6be-8f12366835cc" (UID: "479718cc-f691-42a3-b6be-8f12366835cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:28.636103 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.635418 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467bdac4-5015-4338-a6b1-c7591c441847-kube-api-access-l947c" pod "467bdac4-5015-4338-a6b1-c7591c441847" (UID: "467bdac4-5015-4338-a6b1-c7591c441847"). InnerVolumeSpecName "kube-api-access-l947c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:28.638678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.638552 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479718cc-f691-42a3-b6be-8f12366835cc-kube-api-access-ffgwf" pod "479718cc-f691-42a3-b6be-8f12366835cc" (UID: "479718cc-f691-42a3-b6be-8f12366835cc"). InnerVolumeSpecName "kube-api-access-ffgwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:28.639706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.639181 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467bdac4-5015-4338-a6b1-c7591c441847-istiod-ca-cert" pod "467bdac4-5015-4338-a6b1-c7591c441847" (UID: "467bdac4-5015-4338-a6b1-c7591c441847"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:28.647013 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.646989 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-istio-data" pod "467bdac4-5015-4338-a6b1-c7591c441847" (UID: "467bdac4-5015-4338-a6b1-c7591c441847"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:28.647336 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.647317 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-data" pod "a3eab170-3ff7-48a5-8f32-9acfcb6db28b" (UID: "a3eab170-3ff7-48a5-8f32-9acfcb6db28b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:28.647654 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.647619 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-workload-certs" pod "479718cc-f691-42a3-b6be-8f12366835cc" (UID: "479718cc-f691-42a3-b6be-8f12366835cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:28.648716 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.648659 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-credential-socket" pod "479718cc-f691-42a3-b6be-8f12366835cc" (UID: "479718cc-f691-42a3-b6be-8f12366835cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:28.649797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.649776 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istiod-ca-cert" pod "a3eab170-3ff7-48a5-8f32-9acfcb6db28b" (UID: "a3eab170-3ff7-48a5-8f32-9acfcb6db28b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:28.650274 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.650252 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-workload-certs" pod "467bdac4-5015-4338-a6b1-c7591c441847" (UID: "467bdac4-5015-4338-a6b1-c7591c441847"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:28.654909 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.654887 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-istio-envoy" pod "467bdac4-5015-4338-a6b1-c7591c441847" (UID: "467bdac4-5015-4338-a6b1-c7591c441847"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:28.655366 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.655344 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-credential-socket" pod "467bdac4-5015-4338-a6b1-c7591c441847" (UID: "467bdac4-5015-4338-a6b1-c7591c441847"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:28.655856 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.655759 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479718cc-f691-42a3-b6be-8f12366835cc-istiod-ca-cert" pod "479718cc-f691-42a3-b6be-8f12366835cc" (UID: "479718cc-f691-42a3-b6be-8f12366835cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:28.656461 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.656440 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-credential-socket" pod "a3eab170-3ff7-48a5-8f32-9acfcb6db28b" (UID: "a3eab170-3ff7-48a5-8f32-9acfcb6db28b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:28.659140 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.658955 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-workload-socket" pod "479718cc-f691-42a3-b6be-8f12366835cc" (UID: "479718cc-f691-42a3-b6be-8f12366835cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:28.668168 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.668141 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/479718cc-f691-42a3-b6be-8f12366835cc-istio-podinfo" pod "479718cc-f691-42a3-b6be-8f12366835cc" (UID: "479718cc-f691-42a3-b6be-8f12366835cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:48:28.668819 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.668781 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467bdac4-5015-4338-a6b1-c7591c441847-istio-token" pod "467bdac4-5015-4338-a6b1-c7591c441847" (UID: "467bdac4-5015-4338-a6b1-c7591c441847"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:28.683092 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.681818 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-kube-api-access-kk6zq" pod "a3eab170-3ff7-48a5-8f32-9acfcb6db28b" (UID: "a3eab170-3ff7-48a5-8f32-9acfcb6db28b"). InnerVolumeSpecName "kube-api-access-kk6zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:28.687276 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.687219 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/467bdac4-5015-4338-a6b1-c7591c441847-istio-podinfo" pod "467bdac4-5015-4338-a6b1-c7591c441847" (UID: "467bdac4-5015-4338-a6b1-c7591c441847"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:48:28.687418 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.687290 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479718cc-f691-42a3-b6be-8f12366835cc-istio-token" pod "479718cc-f691-42a3-b6be-8f12366835cc" (UID: "479718cc-f691-42a3-b6be-8f12366835cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:28.687585 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.687552 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-istio-envoy" pod "479718cc-f691-42a3-b6be-8f12366835cc" (UID: "479718cc-f691-42a3-b6be-8f12366835cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:28.688566 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.688535 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-envoy" pod "a3eab170-3ff7-48a5-8f32-9acfcb6db28b" (UID: "a3eab170-3ff7-48a5-8f32-9acfcb6db28b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:28.698022 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.697976 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-token" pod "a3eab170-3ff7-48a5-8f32-9acfcb6db28b" (UID: "a3eab170-3ff7-48a5-8f32-9acfcb6db28b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:28.700027 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.700000 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:28.731885 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.731848 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffgwf\" (UniqueName: \"kubernetes.io/projected/479718cc-f691-42a3-b6be-8f12366835cc-kube-api-access-ffgwf\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.731885 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.731887 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467bdac4-5015-4338-a6b1-c7591c441847-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.731906 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l947c\" (UniqueName: \"kubernetes.io/projected/467bdac4-5015-4338-a6b1-c7591c441847-kube-api-access-l947c\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.731922 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.731937 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.731955 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479718cc-f691-42a3-b6be-8f12366835cc-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.731969 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.731986 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.732000 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.732014 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.732028 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467bdac4-5015-4338-a6b1-c7591c441847-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.732041 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.732055 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479718cc-f691-42a3-b6be-8f12366835cc-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.732068 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.732083 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.732099 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.732113 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.732127 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732871 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.732141 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732871 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.732155 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732871 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.732169 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bdac4-5015-4338-a6b1-c7591c441847-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732871 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.732183 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467bdac4-5015-4338-a6b1-c7591c441847-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732871 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.732198 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk6zq\" (UniqueName: \"kubernetes.io/projected/a3eab170-3ff7-48a5-8f32-9acfcb6db28b-kube-api-access-kk6zq\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732871 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.732213 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479718cc-f691-42a3-b6be-8f12366835cc-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.732871 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.732226 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479718cc-f691-42a3-b6be-8f12366835cc-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:28.942558 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:28.941780 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:29.079660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.074465 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:29.079660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.078075 4519 generic.go:356] "Generic (PLEG): container finished" podID="ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f" containerID="4c3a3dc9dca290beeaaf3e31c75798d9b3951c6ff8c37e3964934cc8ed5cc71a" exitCode=0 Oct 31 00:48:29.079660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.078218 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" event={"ID":"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f","Type":"ContainerDied","Data":"4c3a3dc9dca290beeaaf3e31c75798d9b3951c6ff8c37e3964934cc8ed5cc71a"} Oct 31 00:48:29.126669 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.122531 4519 generic.go:356] "Generic (PLEG): container finished" podID="467bdac4-5015-4338-a6b1-c7591c441847" containerID="238f611f7c542b7063070f09d96c36047096debb888a7cee60a445f152d69b36" exitCode=0 Oct 31 00:48:29.126669 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.122662 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" event={"ID":"467bdac4-5015-4338-a6b1-c7591c441847","Type":"ContainerDied","Data":"238f611f7c542b7063070f09d96c36047096debb888a7cee60a445f152d69b36"} Oct 31 00:48:29.126669 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.122763 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" event={"ID":"467bdac4-5015-4338-a6b1-c7591c441847","Type":"ContainerDied","Data":"eab607a8926ad29d6f5e4f84d738f4a654d2c1e84e0b3aa9b9d170e845c4d124"} Oct 31 00:48:29.126669 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.122813 4519 scope.go:122] "RemoveContainer" containerID="238f611f7c542b7063070f09d96c36047096debb888a7cee60a445f152d69b36" Oct 31 00:48:29.126669 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.122963 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6" Oct 31 00:48:29.171404 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.171373 4519 generic.go:356] "Generic (PLEG): container finished" podID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerID="6f60b4c9e8622704ba2f3440059aadb872b0bab7cfc5020331323d5f4a0d81f6" exitCode=0 Oct 31 00:48:29.171684 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.171628 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" event={"ID":"a3eab170-3ff7-48a5-8f32-9acfcb6db28b","Type":"ContainerDied","Data":"6f60b4c9e8622704ba2f3440059aadb872b0bab7cfc5020331323d5f4a0d81f6"} Oct 31 00:48:29.171830 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.171816 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" event={"ID":"a3eab170-3ff7-48a5-8f32-9acfcb6db28b","Type":"ContainerDied","Data":"bade3321deff64910dd8f9cc66c529bdbd25e429761d03c2aed460ee1a01ee0b"} Oct 31 00:48:29.172010 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.171997 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6" Oct 31 00:48:29.180731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.178069 4519 generic.go:356] "Generic (PLEG): container finished" podID="479718cc-f691-42a3-b6be-8f12366835cc" containerID="33940ca52a3535c1baf63cfe4dd7870ca997b77a4bc1ae54fd366804a0fda590" exitCode=0 Oct 31 00:48:29.180731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.178315 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" Oct 31 00:48:29.180731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.179716 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" event={"ID":"479718cc-f691-42a3-b6be-8f12366835cc","Type":"ContainerDied","Data":"33940ca52a3535c1baf63cfe4dd7870ca997b77a4bc1ae54fd366804a0fda590"} Oct 31 00:48:29.180731 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.179765 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p" event={"ID":"479718cc-f691-42a3-b6be-8f12366835cc","Type":"ContainerDied","Data":"a371c405e873a6e018a35f45bc41e5c27ec217ec2ec21b31701e79a3f94721e7"} Oct 31 00:48:29.181782 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.181757 4519 scope.go:122] "RemoveContainer" containerID="238f611f7c542b7063070f09d96c36047096debb888a7cee60a445f152d69b36" Oct 31 00:48:29.182073 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:48:29.182046 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238f611f7c542b7063070f09d96c36047096debb888a7cee60a445f152d69b36\": container with ID starting with 238f611f7c542b7063070f09d96c36047096debb888a7cee60a445f152d69b36 not found: ID does not exist" containerID="238f611f7c542b7063070f09d96c36047096debb888a7cee60a445f152d69b36" Oct 31 00:48:29.182148 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.182086 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238f611f7c542b7063070f09d96c36047096debb888a7cee60a445f152d69b36"} err="failed to get container status \"238f611f7c542b7063070f09d96c36047096debb888a7cee60a445f152d69b36\": rpc error: code = NotFound desc = could not find container \"238f611f7c542b7063070f09d96c36047096debb888a7cee60a445f152d69b36\": container with ID starting with 238f611f7c542b7063070f09d96c36047096debb888a7cee60a445f152d69b36 not found: ID does not exist" Oct 31 00:48:29.182148 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.182109 4519 scope.go:122] "RemoveContainer" containerID="6f60b4c9e8622704ba2f3440059aadb872b0bab7cfc5020331323d5f4a0d81f6" Oct 31 00:48:29.202479 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.202451 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6"] Oct 31 00:48:29.213064 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.213016 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-lzqm6"] Oct 31 00:48:29.222461 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.222402 4519 scope.go:122] "RemoveContainer" containerID="6f60b4c9e8622704ba2f3440059aadb872b0bab7cfc5020331323d5f4a0d81f6" Oct 31 00:48:29.222878 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:48:29.222850 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f60b4c9e8622704ba2f3440059aadb872b0bab7cfc5020331323d5f4a0d81f6\": container with ID starting with 6f60b4c9e8622704ba2f3440059aadb872b0bab7cfc5020331323d5f4a0d81f6 not found: ID does not exist" containerID="6f60b4c9e8622704ba2f3440059aadb872b0bab7cfc5020331323d5f4a0d81f6" Oct 31 00:48:29.222964 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.222890 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f60b4c9e8622704ba2f3440059aadb872b0bab7cfc5020331323d5f4a0d81f6"} err="failed to get container status \"6f60b4c9e8622704ba2f3440059aadb872b0bab7cfc5020331323d5f4a0d81f6\": rpc error: code = NotFound desc = could not find container \"6f60b4c9e8622704ba2f3440059aadb872b0bab7cfc5020331323d5f4a0d81f6\": container with ID starting with 6f60b4c9e8622704ba2f3440059aadb872b0bab7cfc5020331323d5f4a0d81f6 not found: ID does not exist" Oct 31 00:48:29.222964 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.222914 4519 scope.go:122] "RemoveContainer" containerID="33940ca52a3535c1baf63cfe4dd7870ca997b77a4bc1ae54fd366804a0fda590" Oct 31 00:48:29.240708 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.240677 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6"] Oct 31 00:48:29.244513 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.243188 4519 scope.go:122] "RemoveContainer" containerID="33940ca52a3535c1baf63cfe4dd7870ca997b77a4bc1ae54fd366804a0fda590" Oct 31 00:48:29.244812 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:48:29.244724 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33940ca52a3535c1baf63cfe4dd7870ca997b77a4bc1ae54fd366804a0fda590\": container with ID starting with 33940ca52a3535c1baf63cfe4dd7870ca997b77a4bc1ae54fd366804a0fda590 not found: ID does not exist" containerID="33940ca52a3535c1baf63cfe4dd7870ca997b77a4bc1ae54fd366804a0fda590" Oct 31 00:48:29.244812 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.244760 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33940ca52a3535c1baf63cfe4dd7870ca997b77a4bc1ae54fd366804a0fda590"} err="failed to get container status \"33940ca52a3535c1baf63cfe4dd7870ca997b77a4bc1ae54fd366804a0fda590\": rpc error: code = NotFound desc = could not find container \"33940ca52a3535c1baf63cfe4dd7870ca997b77a4bc1ae54fd366804a0fda590\": container with ID starting with 33940ca52a3535c1baf63cfe4dd7870ca997b77a4bc1ae54fd366804a0fda590 not found: ID does not exist" Oct 31 00:48:29.246753 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.246736 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-h26w6"] Oct 31 00:48:29.267805 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.267776 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p"] Oct 31 00:48:29.272126 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.272051 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-47f370-openshift-default-6dc8794f9-f4h9p"] Oct 31 00:48:29.353427 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.353404 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" Oct 31 00:48:29.453104 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.453010 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ece35843-f7a2-481a-b854-8b15bfdbb38f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece35843-f7a2-481a-b854-8b15bfdbb38f-maas-api-openshift-ca\") pod \"ece35843-f7a2-481a-b854-8b15bfdbb38f\" (UID: \"ece35843-f7a2-481a-b854-8b15bfdbb38f\") " Oct 31 00:48:29.453104 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.453074 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ece35843-f7a2-481a-b854-8b15bfdbb38f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece35843-f7a2-481a-b854-8b15bfdbb38f-maas-api-xks-ca\") pod \"ece35843-f7a2-481a-b854-8b15bfdbb38f\" (UID: \"ece35843-f7a2-481a-b854-8b15bfdbb38f\") " Oct 31 00:48:29.453341 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.453217 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece35843-f7a2-481a-b854-8b15bfdbb38f-kube-api-access-8wjlc\" (UniqueName: \"kubernetes.io/projected/ece35843-f7a2-481a-b854-8b15bfdbb38f-kube-api-access-8wjlc\") pod \"ece35843-f7a2-481a-b854-8b15bfdbb38f\" (UID: \"ece35843-f7a2-481a-b854-8b15bfdbb38f\") " Oct 31 00:48:29.453442 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.453414 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece35843-f7a2-481a-b854-8b15bfdbb38f-maas-api-openshift-ca" pod "ece35843-f7a2-481a-b854-8b15bfdbb38f" (UID: "ece35843-f7a2-481a-b854-8b15bfdbb38f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:29.453771 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.453750 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece35843-f7a2-481a-b854-8b15bfdbb38f-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:29.455326 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.455303 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece35843-f7a2-481a-b854-8b15bfdbb38f-maas-api-xks-ca" pod "ece35843-f7a2-481a-b854-8b15bfdbb38f" (UID: "ece35843-f7a2-481a-b854-8b15bfdbb38f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:29.455433 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.455346 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece35843-f7a2-481a-b854-8b15bfdbb38f-kube-api-access-8wjlc" pod "ece35843-f7a2-481a-b854-8b15bfdbb38f" (UID: "ece35843-f7a2-481a-b854-8b15bfdbb38f"). InnerVolumeSpecName "kube-api-access-8wjlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:29.554829 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.554790 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wjlc\" (UniqueName: \"kubernetes.io/projected/ece35843-f7a2-481a-b854-8b15bfdbb38f-kube-api-access-8wjlc\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:29.554829 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.554822 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece35843-f7a2-481a-b854-8b15bfdbb38f-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:29.701189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.701154 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:48:29.701368 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.701230 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:29.806621 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.806578 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:29.914175 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:29.914136 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:30.067280 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.053539 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:30.104231 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.104199 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467bdac4-5015-4338-a6b1-c7591c441847" path="/var/lib/kubelet/pods/467bdac4-5015-4338-a6b1-c7591c441847/volumes" Oct 31 00:48:30.104733 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.104718 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="479718cc-f691-42a3-b6be-8f12366835cc" path="/var/lib/kubelet/pods/479718cc-f691-42a3-b6be-8f12366835cc/volumes" Oct 31 00:48:30.105162 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.105148 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" path="/var/lib/kubelet/pods/804eee3f-86ba-4511-a9c2-279edb91cf38/volumes" Oct 31 00:48:30.105900 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.105886 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" path="/var/lib/kubelet/pods/a3eab170-3ff7-48a5-8f32-9acfcb6db28b/volumes" Oct 31 00:48:30.106353 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.106330 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" path="/var/lib/kubelet/pods/afa77fbd-2015-4986-bf98-f4858d87f0bf/volumes" Oct 31 00:48:30.228506 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.228381 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" event={"ID":"ece35843-f7a2-481a-b854-8b15bfdbb38f","Type":"ContainerDied","Data":"2e71fde1c0a3cbd9263b816fb4a97080e0e9efa5f9f2e0a6dac03d9b20abd719"} Oct 31 00:48:30.228774 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.228752 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e71fde1c0a3cbd9263b816fb4a97080e0e9efa5f9f2e0a6dac03d9b20abd719" Oct 31 00:48:30.241620 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.230749 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj" Oct 31 00:48:30.485040 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.485010 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" Oct 31 00:48:30.499588 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.499555 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-maas-api-xks-ca\") pod \"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f\" (UID: \"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f\") " Oct 31 00:48:30.499847 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.499819 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-maas-api-openshift-ca\") pod \"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f\" (UID: \"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f\") " Oct 31 00:48:30.500123 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.500055 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-kube-api-access-zfh6d\" (UniqueName: \"kubernetes.io/projected/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-kube-api-access-zfh6d\") pod \"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f\" (UID: \"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f\") " Oct 31 00:48:30.500812 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.500782 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-maas-api-openshift-ca" pod "ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f" (UID: "ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:30.507472 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.507442 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-maas-api-xks-ca" pod "ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f" (UID: "ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:30.510533 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.510507 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-kube-api-access-zfh6d" pod "ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f" (UID: "ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f"). InnerVolumeSpecName "kube-api-access-zfh6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:30.602043 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.602004 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfh6d\" (UniqueName: \"kubernetes.io/projected/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-kube-api-access-zfh6d\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:30.602043 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.602036 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:30.602043 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.602049 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:30.702028 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.701938 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:48:30.702028 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.702006 4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:30.728511 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.728436 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:30.730137 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.730106 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:48:30.854757 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:30.854729 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:31.292693 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.292577 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" event={"ID":"ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f","Type":"ContainerDied","Data":"98eb211e0ec0b5f6e0bfc8f1a6c498f1f65009b8b0fff49a7014f5b4adf19441"} Oct 31 00:48:31.292942 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.292912 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98eb211e0ec0b5f6e0bfc8f1a6c498f1f65009b8b0fff49a7014f5b4adf19441" Oct 31 00:48:31.293148 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.293132 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw" Oct 31 00:48:31.314711 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.314209 4519 generic.go:356] "Generic (PLEG): container finished" podID="7dd65b23-b72a-4eb4-971f-5f218554d0f2" containerID="8134f364768d468c6959d1f9c3bf701e1f827eb2c4c57268a3872f0688d0e76e" exitCode=0 Oct 31 00:48:31.314711 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.314332 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" event={"ID":"7dd65b23-b72a-4eb4-971f-5f218554d0f2","Type":"ContainerDied","Data":"8134f364768d468c6959d1f9c3bf701e1f827eb2c4c57268a3872f0688d0e76e"} Oct 31 00:48:31.319688 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.319454 4519 generic.go:356] "Generic (PLEG): container finished" podID="fd5bf7df-a3e9-4de3-ac84-e258676c6f92" containerID="e775d68e1dcfbef4cab975ccba7835d1894fed65283422e6ce56825a368ec5e0" exitCode=0 Oct 31 00:48:31.320740 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.320711 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" event={"ID":"fd5bf7df-a3e9-4de3-ac84-e258676c6f92","Type":"ContainerDied","Data":"e775d68e1dcfbef4cab975ccba7835d1894fed65283422e6ce56825a368ec5e0"} Oct 31 00:48:31.321352 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.320988 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" Oct 31 00:48:31.337053 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.337022 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5bf7df-a3e9-4de3-ac84-e258676c6f92-kube-api-access-sw764\" (UniqueName: \"kubernetes.io/projected/fd5bf7df-a3e9-4de3-ac84-e258676c6f92-kube-api-access-sw764\") pod \"fd5bf7df-a3e9-4de3-ac84-e258676c6f92\" (UID: \"fd5bf7df-a3e9-4de3-ac84-e258676c6f92\") " Oct 31 00:48:31.337277 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.337256 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd5bf7df-a3e9-4de3-ac84-e258676c6f92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5bf7df-a3e9-4de3-ac84-e258676c6f92-plugins-config-volume\") pod \"fd5bf7df-a3e9-4de3-ac84-e258676c6f92\" (UID: \"fd5bf7df-a3e9-4de3-ac84-e258676c6f92\") " Oct 31 00:48:31.339748 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.339717 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5bf7df-a3e9-4de3-ac84-e258676c6f92-plugins-config-volume" pod "fd5bf7df-a3e9-4de3-ac84-e258676c6f92" (UID: "fd5bf7df-a3e9-4de3-ac84-e258676c6f92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:31.340341 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.340324 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5bf7df-a3e9-4de3-ac84-e258676c6f92-kube-api-access-sw764" pod "fd5bf7df-a3e9-4de3-ac84-e258676c6f92" (UID: "fd5bf7df-a3e9-4de3-ac84-e258676c6f92"). InnerVolumeSpecName "kube-api-access-sw764". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:31.354755 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.353555 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" Oct 31 00:48:31.438658 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.438610 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd65b23-b72a-4eb4-971f-5f218554d0f2-kube-api-access-zw8lz\" (UniqueName: \"kubernetes.io/projected/7dd65b23-b72a-4eb4-971f-5f218554d0f2-kube-api-access-zw8lz\") pod \"7dd65b23-b72a-4eb4-971f-5f218554d0f2\" (UID: \"7dd65b23-b72a-4eb4-971f-5f218554d0f2\") " Oct 31 00:48:31.438898 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.438884 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dd65b23-b72a-4eb4-971f-5f218554d0f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd65b23-b72a-4eb4-971f-5f218554d0f2-plugins-config-volume\") pod \"7dd65b23-b72a-4eb4-971f-5f218554d0f2\" (UID: \"7dd65b23-b72a-4eb4-971f-5f218554d0f2\") " Oct 31 00:48:31.439215 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.439189 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd65b23-b72a-4eb4-971f-5f218554d0f2-plugins-config-volume" pod "7dd65b23-b72a-4eb4-971f-5f218554d0f2" (UID: "7dd65b23-b72a-4eb4-971f-5f218554d0f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:31.439293 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.439203 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw764\" (UniqueName: \"kubernetes.io/projected/fd5bf7df-a3e9-4de3-ac84-e258676c6f92-kube-api-access-sw764\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:31.439293 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.439248 4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5bf7df-a3e9-4de3-ac84-e258676c6f92-plugins-config-volume\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:31.440767 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.440732 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd65b23-b72a-4eb4-971f-5f218554d0f2-kube-api-access-zw8lz" pod "7dd65b23-b72a-4eb4-971f-5f218554d0f2" (UID: "7dd65b23-b72a-4eb4-971f-5f218554d0f2"). InnerVolumeSpecName "kube-api-access-zw8lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:31.540346 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.540311 4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dd65b23-b72a-4eb4-971f-5f218554d0f2-plugins-config-volume\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:31.540346 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.540343 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw8lz\" (UniqueName: \"kubernetes.io/projected/7dd65b23-b72a-4eb4-971f-5f218554d0f2-kube-api-access-zw8lz\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:31.545318 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:31.545227 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:32.225751 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:32.225713 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:32.324973 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:32.324937 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:32.328076 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:32.328050 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" Oct 31 00:48:32.328076 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:32.328056 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv" event={"ID":"7dd65b23-b72a-4eb4-971f-5f218554d0f2","Type":"ContainerDied","Data":"5ac8d277cebb97baafab1893856d0343e642ae92aaae1b20b49e0178f107bb93"} Oct 31 00:48:32.328297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:32.328121 4519 scope.go:122] "RemoveContainer" containerID="8134f364768d468c6959d1f9c3bf701e1f827eb2c4c57268a3872f0688d0e76e" Oct 31 00:48:32.329493 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:32.329473 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" event={"ID":"fd5bf7df-a3e9-4de3-ac84-e258676c6f92","Type":"ContainerDied","Data":"cb971269ad8e8c3cd1d588ce61b6c2345e5a93445701cd622bddc1502d87646d"} Oct 31 00:48:32.329575 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:32.329513 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp" Oct 31 00:48:32.337867 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:32.337846 4519 scope.go:122] "RemoveContainer" containerID="e775d68e1dcfbef4cab975ccba7835d1894fed65283422e6ce56825a368ec5e0" Oct 31 00:48:32.351710 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:32.351681 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp"] Oct 31 00:48:32.355783 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:32.355753 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-44bc67-7d54bb6c7f-qwrvp"] Oct 31 00:48:32.414018 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:32.409777 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv"] Oct 31 00:48:32.445512 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:32.445478 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-44bc67-6d8698bd4f-7hctv"] Oct 31 00:48:33.133764 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:33.133720 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:34.010367 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:34.010335 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:34.145494 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:34.144377 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dd65b23-b72a-4eb4-971f-5f218554d0f2" path="/var/lib/kubelet/pods/7dd65b23-b72a-4eb4-971f-5f218554d0f2/volumes" Oct 31 00:48:34.145494 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:34.145454 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd5bf7df-a3e9-4de3-ac84-e258676c6f92" path="/var/lib/kubelet/pods/fd5bf7df-a3e9-4de3-ac84-e258676c6f92/volumes" Oct 31 00:48:34.501622 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:34.501586 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" podUID="a1a22d4f-183e-4d55-b8f5-71c4d30736bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:48:34.739970 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:34.739938 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:35.129556 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:35.129520 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:35.234678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:35.234063 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:35.331363 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:35.331333 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:35.512033 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:35.511998 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw"] Oct 31 00:48:35.512332 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:35.512300 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" podUID="2ec76a17-5431-4756-a824-862c354595c8" containerName="istio-proxy" containerID="cri-o://9720872459b7ecdaea62f99eee5a354f8e2b8c4ba7303d20000cf0d87249aae6" gracePeriod=30 Oct 31 00:48:35.526066 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:35.526034 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:35.530995 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:35.530967 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g"] Oct 31 00:48:35.531339 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:35.531314 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" podUID="f6b39483-2371-4570-843b-c4421cad360f" containerName="istio-proxy" containerID="cri-o://e6a7506007587f767fe365ecda819b3de4bf0d1cebc74989349a037681156d81" gracePeriod=30 Oct 31 00:48:35.639673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:35.639626 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:35.643836 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:35.643796 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf"] Oct 31 00:48:35.644231 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:35.644202 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" podUID="4320a978-89eb-4601-a10e-028e2549b7dd" containerName="istio-proxy" containerID="cri-o://982cddf7e227e6de5be5793dfd554fe5faf543e77f92aba25cfc581cf4c3957b" gracePeriod=30 Oct 31 00:48:35.651266 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:35.651030 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz"] Oct 31 00:48:35.651408 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:35.651294 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" podUID="393b6b08-821f-4788-9da8-230f14d4e867" containerName="istio-proxy" containerID="cri-o://11929329d7d3a797dbf1b9bcbe81de40f92972aa62858c5087ef6f0f6984d00b" gracePeriod=30 Oct 31 00:48:35.727505 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:35.727465 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:35.909769 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:35.901818 4519 patch_prober.go:29] interesting pod/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:48:35.909769 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:35.901880 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" podUID="4320a978-89eb-4601-a10e-028e2549b7dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:36.350878 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:36.350839 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:37.423659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:37.423564 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:37.672816 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:37.672780 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:38.126830 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:38.126790 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:38.216839 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:38.216778 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:38.317175 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:38.317141 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:38.418992 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:38.418876 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:38.493423 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:38.493335 4519 patch_prober.go:29] interesting pod/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:48:38.493933 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:38.493432 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" podUID="f6b39483-2371-4570-843b-c4421cad360f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:39.113171 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:39.113137 4519 patch_prober.go:29] interesting pod/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:48:39.113326 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:39.113191 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" podUID="393b6b08-821f-4788-9da8-230f14d4e867" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:39.142447 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:39.142414 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:39.449987 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:39.449864 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:39.666428 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:39.666395 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:39.715561 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:39.715491 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" Oct 31 00:48:39.777997 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:39.777964 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:39.795278 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:39.795238 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a22d4f-183e-4d55-b8f5-71c4d30736bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a22d4f-183e-4d55-b8f5-71c4d30736bb-plugins-config-volume\") pod \"a1a22d4f-183e-4d55-b8f5-71c4d30736bb\" (UID: \"a1a22d4f-183e-4d55-b8f5-71c4d30736bb\") " Oct 31 00:48:39.795678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:39.795631 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a22d4f-183e-4d55-b8f5-71c4d30736bb-kube-api-access-gbj59\" (UniqueName: \"kubernetes.io/projected/a1a22d4f-183e-4d55-b8f5-71c4d30736bb-kube-api-access-gbj59\") pod \"a1a22d4f-183e-4d55-b8f5-71c4d30736bb\" (UID: \"a1a22d4f-183e-4d55-b8f5-71c4d30736bb\") " Oct 31 00:48:39.799829 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:39.798726 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a22d4f-183e-4d55-b8f5-71c4d30736bb-plugins-config-volume" pod "a1a22d4f-183e-4d55-b8f5-71c4d30736bb" (UID: "a1a22d4f-183e-4d55-b8f5-71c4d30736bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:39.805790 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:39.805751 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a22d4f-183e-4d55-b8f5-71c4d30736bb-kube-api-access-gbj59" pod "a1a22d4f-183e-4d55-b8f5-71c4d30736bb" (UID: "a1a22d4f-183e-4d55-b8f5-71c4d30736bb"). InnerVolumeSpecName "kube-api-access-gbj59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:39.840140 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:39.832394 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld"] Oct 31 00:48:39.887684 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:39.879666 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:39.901811 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:39.901775 4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a22d4f-183e-4d55-b8f5-71c4d30736bb-plugins-config-volume\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:39.956814 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:39.920866 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbj59\" (UniqueName: \"kubernetes.io/projected/a1a22d4f-183e-4d55-b8f5-71c4d30736bb-kube-api-access-gbj59\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:40.015873 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:39.958924 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2"] Oct 31 00:48:40.016114 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:39.972655 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024561 4519 cpu_manager.go:414] "containerMap: removing container" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024598 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024620 4519 cpu_manager.go:414] "containerMap: removing container" podUID="fd5bf7df-a3e9-4de3-ac84-e258676c6f92" containerName="payload-pre-processing" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024629 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd5bf7df-a3e9-4de3-ac84-e258676c6f92" containerName="payload-pre-processing" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024657 4519 cpu_manager.go:414] "containerMap: removing container" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024665 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024677 4519 cpu_manager.go:414] "containerMap: removing container" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024685 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024707 4519 cpu_manager.go:414] "containerMap: removing container" podUID="ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f" containerName="revoke-keys" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024715 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f" containerName="revoke-keys" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024739 4519 cpu_manager.go:414] "containerMap: removing container" podUID="a1a22d4f-183e-4d55-b8f5-71c4d30736bb" containerName="payload-processing" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024748 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a22d4f-183e-4d55-b8f5-71c4d30736bb" containerName="payload-processing" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024762 4519 cpu_manager.go:414] "containerMap: removing container" podUID="467bdac4-5015-4338-a6b1-c7591c441847" containerName="istio-proxy" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024770 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467bdac4-5015-4338-a6b1-c7591c441847" containerName="istio-proxy" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024806 4519 cpu_manager.go:414] "containerMap: removing container" podUID="ece35843-f7a2-481a-b854-8b15bfdbb38f" containerName="revoke-keys" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024814 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ece35843-f7a2-481a-b854-8b15bfdbb38f" containerName="revoke-keys" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024834 4519 cpu_manager.go:414] "containerMap: removing container" podUID="7dd65b23-b72a-4eb4-971f-5f218554d0f2" containerName="payload-processing" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024842 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd65b23-b72a-4eb4-971f-5f218554d0f2" containerName="payload-processing" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024853 4519 cpu_manager.go:414] "containerMap: removing container" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024860 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024871 4519 cpu_manager.go:414] "containerMap: removing container" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024880 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024894 4519 cpu_manager.go:414] "containerMap: removing container" podUID="479718cc-f691-42a3-b6be-8f12366835cc" containerName="istio-proxy" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024901 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479718cc-f691-42a3-b6be-8f12366835cc" containerName="istio-proxy" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024911 4519 cpu_manager.go:414] "containerMap: removing container" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerName="istio-proxy" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.024920 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerName="istio-proxy" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.025034 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ece35843-f7a2-481a-b854-8b15bfdbb38f" containerName="revoke-keys" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.025049 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd65b23-b72a-4eb4-971f-5f218554d0f2" containerName="payload-processing" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.025060 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="804eee3f-86ba-4511-a9c2-279edb91cf38" containerName="main" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.025069 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a6c2be-3886-4f44-a65a-7eef406c0a45" containerName="main" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.025077 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="479718cc-f691-42a3-b6be-8f12366835cc" containerName="istio-proxy" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.025086 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3eab170-3ff7-48a5-8f32-9acfcb6db28b" containerName="istio-proxy" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.025095 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="886c14ce-3e7b-491b-af66-4a7c9b119861" containerName="main" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.025103 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd5bf7df-a3e9-4de3-ac84-e258676c6f92" containerName="payload-pre-processing" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.025109 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f863a10-a9a8-4894-ac86-2f057371b472" containerName="main" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.025117 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa77fbd-2015-4986-bf98-f4858d87f0bf" containerName="main" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.025127 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f" containerName="revoke-keys" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.025136 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a22d4f-183e-4d55-b8f5-71c4d30736bb" containerName="payload-processing" Oct 31 00:48:40.102180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.025145 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="467bdac4-5015-4338-a6b1-c7591c441847" containerName="istio-proxy" Oct 31 00:48:40.122749 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.121063 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2"] Oct 31 00:48:40.123815 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.123106 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" Oct 31 00:48:40.127801 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.126319 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a90c24bc-gpsg2\" (UID: \"a10862b6-342b-4eb3-9c1a-5ed66b45ba02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" Oct 31 00:48:40.127801 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.126420 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk5fm\" (UniqueName: \"kubernetes.io/projected/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-kube-api-access-zk5fm\") pod \"maas-api-revoke-sub-a90c24bc-gpsg2\" (UID: \"a10862b6-342b-4eb3-9c1a-5ed66b45ba02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" Oct 31 00:48:40.127801 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.126477 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a90c24bc-gpsg2\" (UID: \"a10862b6-342b-4eb3-9c1a-5ed66b45ba02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" Oct 31 00:48:40.137369 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.137341 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:48:40.258661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.257127 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:40.258661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.257972 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk5fm\" (UniqueName: \"kubernetes.io/projected/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-kube-api-access-zk5fm\") pod \"maas-api-revoke-sub-a90c24bc-gpsg2\" (UID: \"a10862b6-342b-4eb3-9c1a-5ed66b45ba02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" Oct 31 00:48:40.258661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.258043 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a90c24bc-gpsg2\" (UID: \"a10862b6-342b-4eb3-9c1a-5ed66b45ba02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" Oct 31 00:48:40.258661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.258284 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a90c24bc-gpsg2\" (UID: \"a10862b6-342b-4eb3-9c1a-5ed66b45ba02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" Oct 31 00:48:40.279195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.260720 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:40.279758 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.276053 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a90c24bc-gpsg2\" (UID: \"a10862b6-342b-4eb3-9c1a-5ed66b45ba02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" Oct 31 00:48:40.281157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.281128 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk5fm\" (UniqueName: \"kubernetes.io/projected/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-kube-api-access-zk5fm\") pod \"maas-api-revoke-sub-a90c24bc-gpsg2\" (UID: \"a10862b6-342b-4eb3-9c1a-5ed66b45ba02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" Oct 31 00:48:40.305922 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.300962 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a90c24bc-gpsg2\" (UID: \"a10862b6-342b-4eb3-9c1a-5ed66b45ba02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" Oct 31 00:48:40.333265 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.333158 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7"] Oct 31 00:48:40.339329 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.339292 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7f1496-wq9m7"] Oct 31 00:48:40.375826 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.375793 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:40.472584 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.472552 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:40.510478 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.510436 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" Oct 31 00:48:40.544856 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.544312 4519 generic.go:356] "Generic (PLEG): container finished" podID="a1a22d4f-183e-4d55-b8f5-71c4d30736bb" containerID="d055e0dd2ce7e62f3e3a0436b35d38ee279a102d2476e730b5f6a5bbe4d56b65" exitCode=0 Oct 31 00:48:40.544856 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.544520 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" event={"ID":"a1a22d4f-183e-4d55-b8f5-71c4d30736bb","Type":"ContainerDied","Data":"d055e0dd2ce7e62f3e3a0436b35d38ee279a102d2476e730b5f6a5bbe4d56b65"} Oct 31 00:48:40.544856 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.544600 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" event={"ID":"a1a22d4f-183e-4d55-b8f5-71c4d30736bb","Type":"ContainerDied","Data":"7c21d27e68b2347acb53c28039264b9b94a53c76bcd878a1a30afe8f1a9b1e13"} Oct 31 00:48:40.544856 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.544670 4519 scope.go:122] "RemoveContainer" containerID="d055e0dd2ce7e62f3e3a0436b35d38ee279a102d2476e730b5f6a5bbe4d56b65" Oct 31 00:48:40.545089 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.544936 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm" Oct 31 00:48:40.688073 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.688013 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" containerID="cri-o://a80a3f83d9ccc01363f66ec7f36548b74dbcc02c7cf5c91c0fcac065f45a700f" gracePeriod=44 Oct 31 00:48:40.857324 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.857083 4519 scope.go:122] "RemoveContainer" containerID="d055e0dd2ce7e62f3e3a0436b35d38ee279a102d2476e730b5f6a5bbe4d56b65" Oct 31 00:48:40.858676 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:48:40.857559 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d055e0dd2ce7e62f3e3a0436b35d38ee279a102d2476e730b5f6a5bbe4d56b65\": container with ID starting with d055e0dd2ce7e62f3e3a0436b35d38ee279a102d2476e730b5f6a5bbe4d56b65 not found: ID does not exist" containerID="d055e0dd2ce7e62f3e3a0436b35d38ee279a102d2476e730b5f6a5bbe4d56b65" Oct 31 00:48:40.858676 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.857597 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d055e0dd2ce7e62f3e3a0436b35d38ee279a102d2476e730b5f6a5bbe4d56b65"} err="failed to get container status \"d055e0dd2ce7e62f3e3a0436b35d38ee279a102d2476e730b5f6a5bbe4d56b65\": rpc error: code = NotFound desc = could not find container \"d055e0dd2ce7e62f3e3a0436b35d38ee279a102d2476e730b5f6a5bbe4d56b65\": container with ID starting with d055e0dd2ce7e62f3e3a0436b35d38ee279a102d2476e730b5f6a5bbe4d56b65 not found: ID does not exist" Oct 31 00:48:40.872559 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.871518 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm"] Oct 31 00:48:40.881549 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.881228 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7f1496-8496858f5-gzkmm"] Oct 31 00:48:40.916901 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.916798 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:40.946279 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:40.945951 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:48:41.007929 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.007903 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-workload-certs\") pod \"f6b39483-2371-4570-843b-c4421cad360f\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " Oct 31 00:48:41.008156 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.008136 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b39483-2371-4570-843b-c4421cad360f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b39483-2371-4570-843b-c4421cad360f-istiod-ca-cert\") pod \"f6b39483-2371-4570-843b-c4421cad360f\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " Oct 31 00:48:41.008290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.008272 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-workload-socket\") pod \"f6b39483-2371-4570-843b-c4421cad360f\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " Oct 31 00:48:41.008807 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.008786 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b39483-2371-4570-843b-c4421cad360f-istio-token\" (UniqueName: \"kubernetes.io/projected/f6b39483-2371-4570-843b-c4421cad360f-istio-token\") pod \"f6b39483-2371-4570-843b-c4421cad360f\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " Oct 31 00:48:41.009107 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.009088 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-credential-socket\") pod \"f6b39483-2371-4570-843b-c4421cad360f\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " Oct 31 00:48:41.009248 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.009231 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-istio-envoy\") pod \"f6b39483-2371-4570-843b-c4421cad360f\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " Oct 31 00:48:41.009368 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.009353 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6b39483-2371-4570-843b-c4421cad360f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b39483-2371-4570-843b-c4421cad360f-istio-podinfo\") pod \"f6b39483-2371-4570-843b-c4421cad360f\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " Oct 31 00:48:41.009494 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.009477 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-istio-data\") pod \"f6b39483-2371-4570-843b-c4421cad360f\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " Oct 31 00:48:41.009606 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.009592 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b39483-2371-4570-843b-c4421cad360f-kube-api-access-2ntpf\" (UniqueName: \"kubernetes.io/projected/f6b39483-2371-4570-843b-c4421cad360f-kube-api-access-2ntpf\") pod \"f6b39483-2371-4570-843b-c4421cad360f\" (UID: \"f6b39483-2371-4570-843b-c4421cad360f\") " Oct 31 00:48:41.010656 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.008458 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-workload-certs" pod "f6b39483-2371-4570-843b-c4421cad360f" (UID: "f6b39483-2371-4570-843b-c4421cad360f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.010779 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.008917 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b39483-2371-4570-843b-c4421cad360f-istiod-ca-cert" pod "f6b39483-2371-4570-843b-c4421cad360f" (UID: "f6b39483-2371-4570-843b-c4421cad360f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:41.010875 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.009331 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-workload-socket" pod "f6b39483-2371-4570-843b-c4421cad360f" (UID: "f6b39483-2371-4570-843b-c4421cad360f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.011384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.011356 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-credential-socket" pod "f6b39483-2371-4570-843b-c4421cad360f" (UID: "f6b39483-2371-4570-843b-c4421cad360f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.013456 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.011415 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-istio-data" pod "f6b39483-2371-4570-843b-c4421cad360f" (UID: "f6b39483-2371-4570-843b-c4421cad360f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.013456 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.012077 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b39483-2371-4570-843b-c4421cad360f-istio-token" pod "f6b39483-2371-4570-843b-c4421cad360f" (UID: "f6b39483-2371-4570-843b-c4421cad360f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:41.016985 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.016871 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b39483-2371-4570-843b-c4421cad360f-kube-api-access-2ntpf" pod "f6b39483-2371-4570-843b-c4421cad360f" (UID: "f6b39483-2371-4570-843b-c4421cad360f"). InnerVolumeSpecName "kube-api-access-2ntpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:41.017365 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.017340 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-istio-envoy" pod "f6b39483-2371-4570-843b-c4421cad360f" (UID: "f6b39483-2371-4570-843b-c4421cad360f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.020618 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.020598 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6b39483-2371-4570-843b-c4421cad360f-istio-podinfo" pod "f6b39483-2371-4570-843b-c4421cad360f" (UID: "f6b39483-2371-4570-843b-c4421cad360f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:48:41.031002 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.030601 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2"] Oct 31 00:48:41.037543 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.037480 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:48:41.045402 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.045321 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:48:41.069617 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.069584 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:48:41.080853 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.080830 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:48:41.111261 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.111221 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec76a17-5431-4756-a824-862c354595c8-kube-api-access-jzvd5\" (UniqueName: \"kubernetes.io/projected/2ec76a17-5431-4756-a824-862c354595c8-kube-api-access-jzvd5\") pod \"2ec76a17-5431-4756-a824-862c354595c8\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " Oct 31 00:48:41.112058 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.112023 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4320a978-89eb-4601-a10e-028e2549b7dd-istio-token\" (UniqueName: \"kubernetes.io/projected/4320a978-89eb-4601-a10e-028e2549b7dd-istio-token\") pod \"4320a978-89eb-4601-a10e-028e2549b7dd\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " Oct 31 00:48:41.112201 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.112074 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-istio-envoy\") pod \"4320a978-89eb-4601-a10e-028e2549b7dd\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " Oct 31 00:48:41.112201 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.112105 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-istio-data\") pod \"2ec76a17-5431-4756-a824-862c354595c8\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " Oct 31 00:48:41.112201 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.112147 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/393b6b08-821f-4788-9da8-230f14d4e867-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393b6b08-821f-4788-9da8-230f14d4e867-istio-podinfo\") pod \"393b6b08-821f-4788-9da8-230f14d4e867\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " Oct 31 00:48:41.112201 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.112186 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393b6b08-821f-4788-9da8-230f14d4e867-istio-token\" (UniqueName: \"kubernetes.io/projected/393b6b08-821f-4788-9da8-230f14d4e867-istio-token\") pod \"393b6b08-821f-4788-9da8-230f14d4e867\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " Oct 31 00:48:41.112450 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.112219 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ec76a17-5431-4756-a824-862c354595c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec76a17-5431-4756-a824-862c354595c8-istio-podinfo\") pod \"2ec76a17-5431-4756-a824-862c354595c8\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " Oct 31 00:48:41.112450 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.112267 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-workload-certs\") pod \"393b6b08-821f-4788-9da8-230f14d4e867\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " Oct 31 00:48:41.112450 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.112293 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-istio-envoy\") pod \"393b6b08-821f-4788-9da8-230f14d4e867\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " Oct 31 00:48:41.112450 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.112322 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393b6b08-821f-4788-9da8-230f14d4e867-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393b6b08-821f-4788-9da8-230f14d4e867-istiod-ca-cert\") pod \"393b6b08-821f-4788-9da8-230f14d4e867\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " Oct 31 00:48:41.112450 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.112403 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-workload-socket\") pod \"2ec76a17-5431-4756-a824-862c354595c8\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " Oct 31 00:48:41.112450 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.112448 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-workload-socket\") pod \"4320a978-89eb-4601-a10e-028e2549b7dd\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " Oct 31 00:48:41.113040 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.113011 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-workload-certs\") pod \"2ec76a17-5431-4756-a824-862c354595c8\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " Oct 31 00:48:41.113157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.113065 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4320a978-89eb-4601-a10e-028e2549b7dd-kube-api-access-926r9\" (UniqueName: \"kubernetes.io/projected/4320a978-89eb-4601-a10e-028e2549b7dd-kube-api-access-926r9\") pod \"4320a978-89eb-4601-a10e-028e2549b7dd\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " Oct 31 00:48:41.113218 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.113156 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4320a978-89eb-4601-a10e-028e2549b7dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4320a978-89eb-4601-a10e-028e2549b7dd-istio-podinfo\") pod \"4320a978-89eb-4601-a10e-028e2549b7dd\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " Oct 31 00:48:41.113218 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.113188 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393b6b08-821f-4788-9da8-230f14d4e867-kube-api-access-5d6qm\" (UniqueName: \"kubernetes.io/projected/393b6b08-821f-4788-9da8-230f14d4e867-kube-api-access-5d6qm\") pod \"393b6b08-821f-4788-9da8-230f14d4e867\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " Oct 31 00:48:41.113312 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.113243 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-credential-socket\") pod \"2ec76a17-5431-4756-a824-862c354595c8\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " Oct 31 00:48:41.113312 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.113294 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-istio-data\") pod \"393b6b08-821f-4788-9da8-230f14d4e867\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " Oct 31 00:48:41.113407 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.113341 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-credential-socket\") pod \"393b6b08-821f-4788-9da8-230f14d4e867\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " Oct 31 00:48:41.113407 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.113401 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-istio-envoy\") pod \"2ec76a17-5431-4756-a824-862c354595c8\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " Oct 31 00:48:41.113496 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.113450 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-credential-socket\") pod \"4320a978-89eb-4601-a10e-028e2549b7dd\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " Oct 31 00:48:41.113542 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.113497 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4320a978-89eb-4601-a10e-028e2549b7dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4320a978-89eb-4601-a10e-028e2549b7dd-istiod-ca-cert\") pod \"4320a978-89eb-4601-a10e-028e2549b7dd\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " Oct 31 00:48:41.113590 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.113551 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-workload-socket\") pod \"393b6b08-821f-4788-9da8-230f14d4e867\" (UID: \"393b6b08-821f-4788-9da8-230f14d4e867\") " Oct 31 00:48:41.113674 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.113591 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ec76a17-5431-4756-a824-862c354595c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec76a17-5431-4756-a824-862c354595c8-istiod-ca-cert\") pod \"2ec76a17-5431-4756-a824-862c354595c8\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " Oct 31 00:48:41.113674 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.113618 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec76a17-5431-4756-a824-862c354595c8-istio-token\" (UniqueName: \"kubernetes.io/projected/2ec76a17-5431-4756-a824-862c354595c8-istio-token\") pod \"2ec76a17-5431-4756-a824-862c354595c8\" (UID: \"2ec76a17-5431-4756-a824-862c354595c8\") " Oct 31 00:48:41.113772 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.113674 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-istio-data\") pod \"4320a978-89eb-4601-a10e-028e2549b7dd\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " Oct 31 00:48:41.113772 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.113712 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-workload-certs\") pod \"4320a978-89eb-4601-a10e-028e2549b7dd\" (UID: \"4320a978-89eb-4601-a10e-028e2549b7dd\") " Oct 31 00:48:41.113772 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.113744 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-workload-certs" pod "393b6b08-821f-4788-9da8-230f14d4e867" (UID: "393b6b08-821f-4788-9da8-230f14d4e867"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.114432 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.114408 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b39483-2371-4570-843b-c4421cad360f-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.114527 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.114437 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.114527 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.114452 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.114527 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.114468 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.114527 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.114480 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b39483-2371-4570-843b-c4421cad360f-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.114527 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.114495 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.114527 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.114508 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ntpf\" (UniqueName: \"kubernetes.io/projected/f6b39483-2371-4570-843b-c4421cad360f-kube-api-access-2ntpf\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.114527 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.114523 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.114900 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.114539 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b39483-2371-4570-843b-c4421cad360f-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.114900 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.114553 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b39483-2371-4570-843b-c4421cad360f-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.114900 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.114788 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-workload-certs" pod "4320a978-89eb-4601-a10e-028e2549b7dd" (UID: "4320a978-89eb-4601-a10e-028e2549b7dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.115051 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.114996 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-credential-socket" pod "393b6b08-821f-4788-9da8-230f14d4e867" (UID: "393b6b08-821f-4788-9da8-230f14d4e867"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.115366 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.115337 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-workload-socket" pod "393b6b08-821f-4788-9da8-230f14d4e867" (UID: "393b6b08-821f-4788-9da8-230f14d4e867"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.115619 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.115591 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-istio-data" pod "393b6b08-821f-4788-9da8-230f14d4e867" (UID: "393b6b08-821f-4788-9da8-230f14d4e867"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.117258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.116058 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-credential-socket" pod "4320a978-89eb-4601-a10e-028e2549b7dd" (UID: "4320a978-89eb-4601-a10e-028e2549b7dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.117258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.116265 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-istio-data" pod "2ec76a17-5431-4756-a824-862c354595c8" (UID: "2ec76a17-5431-4756-a824-862c354595c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.117258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.116281 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-credential-socket" pod "2ec76a17-5431-4756-a824-862c354595c8" (UID: "2ec76a17-5431-4756-a824-862c354595c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.117258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.116793 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4320a978-89eb-4601-a10e-028e2549b7dd-istiod-ca-cert" pod "4320a978-89eb-4601-a10e-028e2549b7dd" (UID: "4320a978-89eb-4601-a10e-028e2549b7dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:41.117258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.116857 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-istio-envoy" pod "393b6b08-821f-4788-9da8-230f14d4e867" (UID: "393b6b08-821f-4788-9da8-230f14d4e867"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.117258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.117103 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-workload-socket" pod "4320a978-89eb-4601-a10e-028e2549b7dd" (UID: "4320a978-89eb-4601-a10e-028e2549b7dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.118021 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.117706 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-workload-socket" pod "2ec76a17-5431-4756-a824-862c354595c8" (UID: "2ec76a17-5431-4756-a824-862c354595c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.119058 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.118994 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-workload-certs" pod "2ec76a17-5431-4756-a824-862c354595c8" (UID: "2ec76a17-5431-4756-a824-862c354595c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.119261 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.119208 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-istio-data" pod "4320a978-89eb-4601-a10e-028e2549b7dd" (UID: "4320a978-89eb-4601-a10e-028e2549b7dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.119553 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.119514 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec76a17-5431-4756-a824-862c354595c8-istiod-ca-cert" pod "2ec76a17-5431-4756-a824-862c354595c8" (UID: "2ec76a17-5431-4756-a824-862c354595c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:41.120119 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.120092 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393b6b08-821f-4788-9da8-230f14d4e867-istiod-ca-cert" pod "393b6b08-821f-4788-9da8-230f14d4e867" (UID: "393b6b08-821f-4788-9da8-230f14d4e867"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:41.121420 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.121374 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4320a978-89eb-4601-a10e-028e2549b7dd-istio-podinfo" pod "4320a978-89eb-4601-a10e-028e2549b7dd" (UID: "4320a978-89eb-4601-a10e-028e2549b7dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:48:41.121519 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.121478 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-istio-envoy" pod "4320a978-89eb-4601-a10e-028e2549b7dd" (UID: "4320a978-89eb-4601-a10e-028e2549b7dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.121781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.121757 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4320a978-89eb-4601-a10e-028e2549b7dd-kube-api-access-926r9" pod "4320a978-89eb-4601-a10e-028e2549b7dd" (UID: "4320a978-89eb-4601-a10e-028e2549b7dd"). InnerVolumeSpecName "kube-api-access-926r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:41.121962 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.121929 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/393b6b08-821f-4788-9da8-230f14d4e867-istio-podinfo" pod "393b6b08-821f-4788-9da8-230f14d4e867" (UID: "393b6b08-821f-4788-9da8-230f14d4e867"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:48:41.122076 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.122024 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec76a17-5431-4756-a824-862c354595c8-istio-token" pod "2ec76a17-5431-4756-a824-862c354595c8" (UID: "2ec76a17-5431-4756-a824-862c354595c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:41.122680 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.122621 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393b6b08-821f-4788-9da8-230f14d4e867-istio-token" pod "393b6b08-821f-4788-9da8-230f14d4e867" (UID: "393b6b08-821f-4788-9da8-230f14d4e867"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:41.123489 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.123464 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4320a978-89eb-4601-a10e-028e2549b7dd-istio-token" pod "4320a978-89eb-4601-a10e-028e2549b7dd" (UID: "4320a978-89eb-4601-a10e-028e2549b7dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:41.127093 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.127066 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393b6b08-821f-4788-9da8-230f14d4e867-kube-api-access-5d6qm" pod "393b6b08-821f-4788-9da8-230f14d4e867" (UID: "393b6b08-821f-4788-9da8-230f14d4e867"). InnerVolumeSpecName "kube-api-access-5d6qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:41.127093 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.127073 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ec76a17-5431-4756-a824-862c354595c8-istio-podinfo" pod "2ec76a17-5431-4756-a824-862c354595c8" (UID: "2ec76a17-5431-4756-a824-862c354595c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:48:41.127207 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.127099 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec76a17-5431-4756-a824-862c354595c8-kube-api-access-jzvd5" pod "2ec76a17-5431-4756-a824-862c354595c8" (UID: "2ec76a17-5431-4756-a824-862c354595c8"). InnerVolumeSpecName "kube-api-access-jzvd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:41.128579 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.128560 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-istio-envoy" pod "2ec76a17-5431-4756-a824-862c354595c8" (UID: "2ec76a17-5431-4756-a824-862c354595c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.216074 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.215816 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-home\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-home\") pod \"e27be156-a2bf-407a-a15c-b348f3e87105\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " Oct 31 00:48:41.216074 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.215991 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-dshm\") pod \"e27be156-a2bf-407a-a15c-b348f3e87105\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " Oct 31 00:48:41.216416 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.216388 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e27be156-a2bf-407a-a15c-b348f3e87105-tls-certs\" (UniqueName: \"kubernetes.io/secret/e27be156-a2bf-407a-a15c-b348f3e87105-tls-certs\") pod \"e27be156-a2bf-407a-a15c-b348f3e87105\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " Oct 31 00:48:41.216416 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.216444 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27be156-a2bf-407a-a15c-b348f3e87105-kube-api-access-7ssq9\" (UniqueName: \"kubernetes.io/projected/e27be156-a2bf-407a-a15c-b348f3e87105-kube-api-access-7ssq9\") pod \"e27be156-a2bf-407a-a15c-b348f3e87105\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " Oct 31 00:48:41.216692 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.216582 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-model-cache\") pod \"e27be156-a2bf-407a-a15c-b348f3e87105\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " Oct 31 00:48:41.217249 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.217219 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-home" pod "e27be156-a2bf-407a-a15c-b348f3e87105" (UID: "e27be156-a2bf-407a-a15c-b348f3e87105"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.217591 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.217566 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-tmp-dir\") pod \"e27be156-a2bf-407a-a15c-b348f3e87105\" (UID: \"e27be156-a2bf-407a-a15c-b348f3e87105\") " Oct 31 00:48:41.217753 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.217713 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-model-cache" pod "e27be156-a2bf-407a-a15c-b348f3e87105" (UID: "e27be156-a2bf-407a-a15c-b348f3e87105"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218124 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-tmp-dir" pod "e27be156-a2bf-407a-a15c-b348f3e87105" (UID: "e27be156-a2bf-407a-a15c-b348f3e87105"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218268 4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-model-cache\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218286 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4320a978-89eb-4601-a10e-028e2549b7dd-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218299 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218313 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218326 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393b6b08-821f-4788-9da8-230f14d4e867-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218340 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393b6b08-821f-4788-9da8-230f14d4e867-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218353 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec76a17-5431-4756-a824-862c354595c8-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218367 4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-tmp-dir\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218379 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218392 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393b6b08-821f-4788-9da8-230f14d4e867-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218405 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218419 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218435 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218448 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-926r9\" (UniqueName: \"kubernetes.io/projected/4320a978-89eb-4601-a10e-028e2549b7dd-kube-api-access-926r9\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218462 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4320a978-89eb-4601-a10e-028e2549b7dd-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218479 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d6qm\" (UniqueName: \"kubernetes.io/projected/393b6b08-821f-4788-9da8-230f14d4e867-kube-api-access-5d6qm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218493 4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-home\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218506 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218520 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218533 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218546 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec76a17-5431-4756-a824-862c354595c8-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218559 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218571 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4320a978-89eb-4601-a10e-028e2549b7dd-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218587 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b6b08-821f-4788-9da8-230f14d4e867-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218599 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec76a17-5431-4756-a824-862c354595c8-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218612 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec76a17-5431-4756-a824-862c354595c8-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218626 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218657 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4320a978-89eb-4601-a10e-028e2549b7dd-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.218706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.218670 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzvd5\" (UniqueName: \"kubernetes.io/projected/2ec76a17-5431-4756-a824-862c354595c8-kube-api-access-jzvd5\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.220942 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.220915 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e27be156-a2bf-407a-a15c-b348f3e87105-tls-certs" pod "e27be156-a2bf-407a-a15c-b348f3e87105" (UID: "e27be156-a2bf-407a-a15c-b348f3e87105"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:41.221434 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.221403 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27be156-a2bf-407a-a15c-b348f3e87105-kube-api-access-7ssq9" pod "e27be156-a2bf-407a-a15c-b348f3e87105" (UID: "e27be156-a2bf-407a-a15c-b348f3e87105"). InnerVolumeSpecName "kube-api-access-7ssq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:41.222573 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.222547 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-dshm" pod "e27be156-a2bf-407a-a15c-b348f3e87105" (UID: "e27be156-a2bf-407a-a15c-b348f3e87105"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:41.321931 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.321843 4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e27be156-a2bf-407a-a15c-b348f3e87105-dshm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.322161 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.322140 4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e27be156-a2bf-407a-a15c-b348f3e87105-tls-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.322302 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.322286 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ssq9\" (UniqueName: \"kubernetes.io/projected/e27be156-a2bf-407a-a15c-b348f3e87105-kube-api-access-7ssq9\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:41.511299 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.511256 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:41.552247 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.552208 4519 generic.go:356] "Generic (PLEG): container finished" podID="e27be156-a2bf-407a-a15c-b348f3e87105" containerID="a80a3f83d9ccc01363f66ec7f36548b74dbcc02c7cf5c91c0fcac065f45a700f" exitCode=0 Oct 31 00:48:41.552438 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.552274 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" Oct 31 00:48:41.552438 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.552291 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" event={"ID":"e27be156-a2bf-407a-a15c-b348f3e87105","Type":"ContainerDied","Data":"a80a3f83d9ccc01363f66ec7f36548b74dbcc02c7cf5c91c0fcac065f45a700f"} Oct 31 00:48:41.552438 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.552340 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb" event={"ID":"e27be156-a2bf-407a-a15c-b348f3e87105","Type":"ContainerDied","Data":"69ce29f94d578e274a4b54c1d20ddfdc44537ea5d667390a2f9aa14b349a320e"} Oct 31 00:48:41.552438 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.552359 4519 scope.go:122] "RemoveContainer" containerID="a80a3f83d9ccc01363f66ec7f36548b74dbcc02c7cf5c91c0fcac065f45a700f" Oct 31 00:48:41.553816 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.553781 4519 generic.go:356] "Generic (PLEG): container finished" podID="393b6b08-821f-4788-9da8-230f14d4e867" containerID="11929329d7d3a797dbf1b9bcbe81de40f92972aa62858c5087ef6f0f6984d00b" exitCode=0 Oct 31 00:48:41.553897 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.553861 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" Oct 31 00:48:41.553897 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.553882 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" event={"ID":"393b6b08-821f-4788-9da8-230f14d4e867","Type":"ContainerDied","Data":"11929329d7d3a797dbf1b9bcbe81de40f92972aa62858c5087ef6f0f6984d00b"} Oct 31 00:48:41.554007 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.553973 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz" event={"ID":"393b6b08-821f-4788-9da8-230f14d4e867","Type":"ContainerDied","Data":"c7c7ba4ed1f3444af753ddeb3ebe6472e1655bc1f5cff105595e5d6d99b26cb4"} Oct 31 00:48:41.556459 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.556435 4519 generic.go:356] "Generic (PLEG): container finished" podID="f6b39483-2371-4570-843b-c4421cad360f" containerID="e6a7506007587f767fe365ecda819b3de4bf0d1cebc74989349a037681156d81" exitCode=0 Oct 31 00:48:41.556584 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.556559 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" event={"ID":"f6b39483-2371-4570-843b-c4421cad360f","Type":"ContainerDied","Data":"e6a7506007587f767fe365ecda819b3de4bf0d1cebc74989349a037681156d81"} Oct 31 00:48:41.556657 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.556618 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" event={"ID":"f6b39483-2371-4570-843b-c4421cad360f","Type":"ContainerDied","Data":"b2e6208484d8799c5522bc1a8cac1707c664ad379b820f957e805f78226de834"} Oct 31 00:48:41.556720 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.556571 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g" Oct 31 00:48:41.563851 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.563826 4519 generic.go:356] "Generic (PLEG): container finished" podID="4320a978-89eb-4601-a10e-028e2549b7dd" containerID="982cddf7e227e6de5be5793dfd554fe5faf543e77f92aba25cfc581cf4c3957b" exitCode=0 Oct 31 00:48:41.563969 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.563881 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" Oct 31 00:48:41.563969 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.563909 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" event={"ID":"4320a978-89eb-4601-a10e-028e2549b7dd","Type":"ContainerDied","Data":"982cddf7e227e6de5be5793dfd554fe5faf543e77f92aba25cfc581cf4c3957b"} Oct 31 00:48:41.563969 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.563961 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf" event={"ID":"4320a978-89eb-4601-a10e-028e2549b7dd","Type":"ContainerDied","Data":"ff31bfc0b74f7353275354ca9a4a850a6f028735f4393a48495f168e12a480b0"} Oct 31 00:48:41.564140 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.564117 4519 scope.go:122] "RemoveContainer" containerID="a80a3f83d9ccc01363f66ec7f36548b74dbcc02c7cf5c91c0fcac065f45a700f" Oct 31 00:48:41.564683 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:48:41.564659 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80a3f83d9ccc01363f66ec7f36548b74dbcc02c7cf5c91c0fcac065f45a700f\": container with ID starting with a80a3f83d9ccc01363f66ec7f36548b74dbcc02c7cf5c91c0fcac065f45a700f not found: ID does not exist" containerID="a80a3f83d9ccc01363f66ec7f36548b74dbcc02c7cf5c91c0fcac065f45a700f" Oct 31 00:48:41.564753 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.564695 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80a3f83d9ccc01363f66ec7f36548b74dbcc02c7cf5c91c0fcac065f45a700f"} err="failed to get container status \"a80a3f83d9ccc01363f66ec7f36548b74dbcc02c7cf5c91c0fcac065f45a700f\": rpc error: code = NotFound desc = could not find container \"a80a3f83d9ccc01363f66ec7f36548b74dbcc02c7cf5c91c0fcac065f45a700f\": container with ID starting with a80a3f83d9ccc01363f66ec7f36548b74dbcc02c7cf5c91c0fcac065f45a700f not found: ID does not exist" Oct 31 00:48:41.564753 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.564719 4519 scope.go:122] "RemoveContainer" containerID="11929329d7d3a797dbf1b9bcbe81de40f92972aa62858c5087ef6f0f6984d00b" Oct 31 00:48:41.566807 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.566627 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" event={"ID":"a10862b6-342b-4eb3-9c1a-5ed66b45ba02","Type":"ContainerStarted","Data":"f971c321ea2f0fb242f5b6c007cf5bf3ed3b6e61c409e0f807d8445c5968ef3a"} Oct 31 00:48:41.566807 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.566703 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" event={"ID":"a10862b6-342b-4eb3-9c1a-5ed66b45ba02","Type":"ContainerStarted","Data":"896f4be0e8480a21f8d733e5ee6a8628d91da422f37de86369379b9bd98f0b82"} Oct 31 00:48:41.568866 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.568828 4519 generic.go:356] "Generic (PLEG): container finished" podID="2ec76a17-5431-4756-a824-862c354595c8" containerID="9720872459b7ecdaea62f99eee5a354f8e2b8c4ba7303d20000cf0d87249aae6" exitCode=0 Oct 31 00:48:41.568866 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.568865 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" event={"ID":"2ec76a17-5431-4756-a824-862c354595c8","Type":"ContainerDied","Data":"9720872459b7ecdaea62f99eee5a354f8e2b8c4ba7303d20000cf0d87249aae6"} Oct 31 00:48:41.569036 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.568900 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" event={"ID":"2ec76a17-5431-4756-a824-862c354595c8","Type":"ContainerDied","Data":"b7202743eaf85508f06d19ca86ae0e8b1111f342b6b9a7c08c59cbe3b46e23d7"} Oct 31 00:48:41.569036 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.568912 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw" Oct 31 00:48:41.579569 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.579277 4519 scope.go:122] "RemoveContainer" containerID="11929329d7d3a797dbf1b9bcbe81de40f92972aa62858c5087ef6f0f6984d00b" Oct 31 00:48:41.579803 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:48:41.579780 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11929329d7d3a797dbf1b9bcbe81de40f92972aa62858c5087ef6f0f6984d00b\": container with ID starting with 11929329d7d3a797dbf1b9bcbe81de40f92972aa62858c5087ef6f0f6984d00b not found: ID does not exist" containerID="11929329d7d3a797dbf1b9bcbe81de40f92972aa62858c5087ef6f0f6984d00b" Oct 31 00:48:41.579963 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.579938 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11929329d7d3a797dbf1b9bcbe81de40f92972aa62858c5087ef6f0f6984d00b"} err="failed to get container status \"11929329d7d3a797dbf1b9bcbe81de40f92972aa62858c5087ef6f0f6984d00b\": rpc error: code = NotFound desc = could not find container \"11929329d7d3a797dbf1b9bcbe81de40f92972aa62858c5087ef6f0f6984d00b\": container with ID starting with 11929329d7d3a797dbf1b9bcbe81de40f92972aa62858c5087ef6f0f6984d00b not found: ID does not exist" Oct 31 00:48:41.580065 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.580048 4519 scope.go:122] "RemoveContainer" containerID="e6a7506007587f767fe365ecda819b3de4bf0d1cebc74989349a037681156d81" Oct 31 00:48:41.604072 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.604039 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz"] Oct 31 00:48:41.610169 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.610134 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-445gz"] Oct 31 00:48:41.616484 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.616464 4519 scope.go:122] "RemoveContainer" containerID="e6a7506007587f767fe365ecda819b3de4bf0d1cebc74989349a037681156d81" Oct 31 00:48:41.616867 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:48:41.616844 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6a7506007587f767fe365ecda819b3de4bf0d1cebc74989349a037681156d81\": container with ID starting with e6a7506007587f767fe365ecda819b3de4bf0d1cebc74989349a037681156d81 not found: ID does not exist" containerID="e6a7506007587f767fe365ecda819b3de4bf0d1cebc74989349a037681156d81" Oct 31 00:48:41.616940 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.616875 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6a7506007587f767fe365ecda819b3de4bf0d1cebc74989349a037681156d81"} err="failed to get container status \"e6a7506007587f767fe365ecda819b3de4bf0d1cebc74989349a037681156d81\": rpc error: code = NotFound desc = could not find container \"e6a7506007587f767fe365ecda819b3de4bf0d1cebc74989349a037681156d81\": container with ID starting with e6a7506007587f767fe365ecda819b3de4bf0d1cebc74989349a037681156d81 not found: ID does not exist" Oct 31 00:48:41.616940 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.616892 4519 scope.go:122] "RemoveContainer" containerID="982cddf7e227e6de5be5793dfd554fe5faf543e77f92aba25cfc581cf4c3957b" Oct 31 00:48:41.626343 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.626326 4519 scope.go:122] "RemoveContainer" containerID="982cddf7e227e6de5be5793dfd554fe5faf543e77f92aba25cfc581cf4c3957b" Oct 31 00:48:41.626625 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:48:41.626606 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"982cddf7e227e6de5be5793dfd554fe5faf543e77f92aba25cfc581cf4c3957b\": container with ID starting with 982cddf7e227e6de5be5793dfd554fe5faf543e77f92aba25cfc581cf4c3957b not found: ID does not exist" containerID="982cddf7e227e6de5be5793dfd554fe5faf543e77f92aba25cfc581cf4c3957b" Oct 31 00:48:41.626681 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.626667 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"982cddf7e227e6de5be5793dfd554fe5faf543e77f92aba25cfc581cf4c3957b"} err="failed to get container status \"982cddf7e227e6de5be5793dfd554fe5faf543e77f92aba25cfc581cf4c3957b\": rpc error: code = NotFound desc = could not find container \"982cddf7e227e6de5be5793dfd554fe5faf543e77f92aba25cfc581cf4c3957b\": container with ID starting with 982cddf7e227e6de5be5793dfd554fe5faf543e77f92aba25cfc581cf4c3957b not found: ID does not exist" Oct 31 00:48:41.626738 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.626683 4519 scope.go:122] "RemoveContainer" containerID="9720872459b7ecdaea62f99eee5a354f8e2b8c4ba7303d20000cf0d87249aae6" Oct 31 00:48:41.633403 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.633379 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g"] Oct 31 00:48:41.637806 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.637786 4519 scope.go:122] "RemoveContainer" containerID="9720872459b7ecdaea62f99eee5a354f8e2b8c4ba7303d20000cf0d87249aae6" Oct 31 00:48:41.638114 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:48:41.638093 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9720872459b7ecdaea62f99eee5a354f8e2b8c4ba7303d20000cf0d87249aae6\": container with ID starting with 9720872459b7ecdaea62f99eee5a354f8e2b8c4ba7303d20000cf0d87249aae6 not found: ID does not exist" containerID="9720872459b7ecdaea62f99eee5a354f8e2b8c4ba7303d20000cf0d87249aae6" Oct 31 00:48:41.638202 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.638126 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9720872459b7ecdaea62f99eee5a354f8e2b8c4ba7303d20000cf0d87249aae6"} err="failed to get container status \"9720872459b7ecdaea62f99eee5a354f8e2b8c4ba7303d20000cf0d87249aae6\": rpc error: code = NotFound desc = could not find container \"9720872459b7ecdaea62f99eee5a354f8e2b8c4ba7303d20000cf0d87249aae6\": container with ID starting with 9720872459b7ecdaea62f99eee5a354f8e2b8c4ba7303d20000cf0d87249aae6 not found: ID does not exist" Oct 31 00:48:41.638276 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.638258 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-5lh4g"] Oct 31 00:48:41.659814 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.659775 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb"] Oct 31 00:48:41.663535 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.663505 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f99ac0/distinct-w0-f99ac0-kserve-77f68668c7-4l8mb"] Oct 31 00:48:41.680229 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.680174 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" podStartSLOduration=2.6801604980000002 podStartE2EDuration="2.680160498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:41.679035982 +0000 UTC m=+917.708216393" watchObservedRunningTime="2026-10-31 00:48:41.680160498 +0000 UTC m=+917.709340889" Oct 31 00:48:41.697452 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.697418 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf"] Oct 31 00:48:41.704427 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.704398 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-44bc67-openshift-default-5b7c5d8f6c-l4bpf"] Oct 31 00:48:41.722150 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.722117 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw"] Oct 31 00:48:41.729267 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:41.729232 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d4a7e-openshift-default-559b45695f-f5tfw"] Oct 31 00:48:42.020071 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:42.020042 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:42.110238 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:42.110176 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07e0154e-7cfa-43d1-b996-65611ea1ef4e" path="/var/lib/kubelet/pods/07e0154e-7cfa-43d1-b996-65611ea1ef4e/volumes" Oct 31 00:48:42.111194 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:42.111168 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec76a17-5431-4756-a824-862c354595c8" path="/var/lib/kubelet/pods/2ec76a17-5431-4756-a824-862c354595c8/volumes" Oct 31 00:48:42.111961 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:42.111925 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="393b6b08-821f-4788-9da8-230f14d4e867" path="/var/lib/kubelet/pods/393b6b08-821f-4788-9da8-230f14d4e867/volumes" Oct 31 00:48:42.113293 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:42.113268 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4320a978-89eb-4601-a10e-028e2549b7dd" path="/var/lib/kubelet/pods/4320a978-89eb-4601-a10e-028e2549b7dd/volumes" Oct 31 00:48:42.113971 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:42.113951 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a22d4f-183e-4d55-b8f5-71c4d30736bb" path="/var/lib/kubelet/pods/a1a22d4f-183e-4d55-b8f5-71c4d30736bb/volumes" Oct 31 00:48:42.114713 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:42.114683 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" path="/var/lib/kubelet/pods/e27be156-a2bf-407a-a15c-b348f3e87105/volumes" Oct 31 00:48:42.115857 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:42.115837 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b39483-2371-4570-843b-c4421cad360f" path="/var/lib/kubelet/pods/f6b39483-2371-4570-843b-c4421cad360f/volumes" Oct 31 00:48:42.580064 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:42.580028 4519 generic.go:356] "Generic (PLEG): container finished" podID="a10862b6-342b-4eb3-9c1a-5ed66b45ba02" containerID="f971c321ea2f0fb242f5b6c007cf5bf3ed3b6e61c409e0f807d8445c5968ef3a" exitCode=0 Oct 31 00:48:42.580239 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:42.580111 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" event={"ID":"a10862b6-342b-4eb3-9c1a-5ed66b45ba02","Type":"ContainerDied","Data":"f971c321ea2f0fb242f5b6c007cf5bf3ed3b6e61c409e0f807d8445c5968ef3a"} Oct 31 00:48:42.658957 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:42.658913 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p"] Oct 31 00:48:42.748921 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:42.748887 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:42.976086 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:42.976000 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:43.763215 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:43.763187 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" Oct 31 00:48:43.861036 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:43.860954 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-maas-api-openshift-ca\") pod \"a10862b6-342b-4eb3-9c1a-5ed66b45ba02\" (UID: \"a10862b6-342b-4eb3-9c1a-5ed66b45ba02\") " Oct 31 00:48:43.861036 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:43.861014 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-kube-api-access-zk5fm\" (UniqueName: \"kubernetes.io/projected/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-kube-api-access-zk5fm\") pod \"a10862b6-342b-4eb3-9c1a-5ed66b45ba02\" (UID: \"a10862b6-342b-4eb3-9c1a-5ed66b45ba02\") " Oct 31 00:48:43.861907 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:43.861134 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-maas-api-xks-ca\") pod \"a10862b6-342b-4eb3-9c1a-5ed66b45ba02\" (UID: \"a10862b6-342b-4eb3-9c1a-5ed66b45ba02\") " Oct 31 00:48:43.862678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:43.862627 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-maas-api-openshift-ca" pod "a10862b6-342b-4eb3-9c1a-5ed66b45ba02" (UID: "a10862b6-342b-4eb3-9c1a-5ed66b45ba02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:43.863951 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:43.863925 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:43.865255 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:43.865229 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-maas-api-xks-ca" pod "a10862b6-342b-4eb3-9c1a-5ed66b45ba02" (UID: "a10862b6-342b-4eb3-9c1a-5ed66b45ba02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:43.865463 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:43.865441 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-kube-api-access-zk5fm" pod "a10862b6-342b-4eb3-9c1a-5ed66b45ba02" (UID: "a10862b6-342b-4eb3-9c1a-5ed66b45ba02"). InnerVolumeSpecName "kube-api-access-zk5fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:43.962330 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:43.962299 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:43.962330 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:43.962327 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk5fm\" (UniqueName: \"kubernetes.io/projected/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-kube-api-access-zk5fm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:43.962530 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:43.962339 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10862b6-342b-4eb3-9c1a-5ed66b45ba02-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:44.171709 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:44.157177 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:44.597738 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:44.597704 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" event={"ID":"a10862b6-342b-4eb3-9c1a-5ed66b45ba02","Type":"ContainerDied","Data":"896f4be0e8480a21f8d733e5ee6a8628d91da422f37de86369379b9bd98f0b82"} Oct 31 00:48:44.597738 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:44.597727 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2" Oct 31 00:48:44.597962 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:44.597751 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="896f4be0e8480a21f8d733e5ee6a8628d91da422f37de86369379b9bd98f0b82" Oct 31 00:48:44.841552 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:44.841512 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" podUID="f76bbc2e-d9b0-4ac4-a09d-91b609760c16" containerName="payload-pre-processing" containerID="cri-o://e8b4494e095cdf344c1f883571ca176a2ed607955140e47b6e1b40963ffb0d6a" gracePeriod=25 Oct 31 00:48:44.980289 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:44.980253 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:45.139829 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:45.139788 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:46.079604 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:46.079569 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:46.965998 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:46.965963 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:47.269130 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:47.269093 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:47.543155 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:48:47.543064 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:48:47.764032 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:47.763998 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:47.839289 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:47.839202 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:47.988176 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:47.985090 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k"] Oct 31 00:48:48.074655 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:48.074153 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:48.147136 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:48.145858 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4"] Oct 31 00:48:48.147136 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:48.146073 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" containerID="cri-o://87d40eed09e0c1f606cb69ac3819cefb70bb9d9e4d5c6d3abbfe3267182ef87c" gracePeriod=30 Oct 31 00:48:48.151560 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:48.150797 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr"] Oct 31 00:48:48.151560 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:48.151011 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" podUID="6d89558b-7909-4d86-9305-720c63771d05" containerName="istio-proxy" containerID="cri-o://6e38faa53ef9860a61bfaf406e80af578c7f05723c01b146004605bf1c0847f9" gracePeriod=30 Oct 31 00:48:48.160763 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:48.158192 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh"] Oct 31 00:48:48.160763 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:48.158382 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" podUID="5498f31e-f243-43b8-992a-56cbccdc239c" containerName="istio-proxy" containerID="cri-o://bd61ba53498b29b792c2194cdbdd2024201a24521590e2e096d86e3479cf1cb9" gracePeriod=30 Oct 31 00:48:48.163944 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:48.163917 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w"] Oct 31 00:48:48.164136 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:48.164113 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" containerID="cri-o://562e889c23398bd92b8c294c6094af7e8ba054dd567fb57b0e7f6c22164de97e" gracePeriod=30 Oct 31 00:48:48.178440 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:48.178409 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:48.666825 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:48.666779 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:49.120233 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:49.120201 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8"] Oct 31 00:48:49.133290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:49.133259 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz"] Oct 31 00:48:49.140932 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:49.140901 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl"] Oct 31 00:48:49.207846 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:49.207804 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:49.454280 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:49.454200 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:50.337250 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:50.337215 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:51.031148 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:51.031111 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:51.152509 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:51.152472 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:51.816535 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:51.816500 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:52.547539 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:52.547504 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:52.720888 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:52.720849 4519 patch_prober.go:29] interesting pod/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:48:52.721067 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:52.720914 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" podUID="5498f31e-f243-43b8-992a-56cbccdc239c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:52.985727 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:52.985685 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" podUID="6b03043a-75b4-404a-800f-d71f33a04108" containerName="payload-processing" containerID="cri-o://7544849631b0f15a08d6e0b17f5c84594a76b78e0111356aebde942dad29b6f4" gracePeriod=25 Oct 31 00:48:53.056379 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.056341 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph"] Oct 31 00:48:53.056684 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.056650 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" podUID="638ee841-3ada-4014-8841-f9da201ea000" containerName="maas-api" containerID="cri-o://7012a352ea43e9db0648a6f00bda576fe5333c4fd993ba9d93ad5783dbc6fe7d" gracePeriod=30 Oct 31 00:48:53.082703 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.082596 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:53.212260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.212230 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:53.357656 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.348214 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:53.408792 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.408719 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj"] Oct 31 00:48:53.469744 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.447500 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f99ac0-4d6xj"] Oct 31 00:48:53.488397 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.488292 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:53.497969 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.497681 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" podUID="6b03043a-75b4-404a-800f-d71f33a04108" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:48:53.605385 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.604624 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:48:53.672291 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.672255 4519 generic.go:356] "Generic (PLEG): container finished" podID="5498f31e-f243-43b8-992a-56cbccdc239c" containerID="bd61ba53498b29b792c2194cdbdd2024201a24521590e2e096d86e3479cf1cb9" exitCode=0 Oct 31 00:48:53.672554 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.672314 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" event={"ID":"5498f31e-f243-43b8-992a-56cbccdc239c","Type":"ContainerDied","Data":"bd61ba53498b29b792c2194cdbdd2024201a24521590e2e096d86e3479cf1cb9"} Oct 31 00:48:53.674718 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.674627 4519 generic.go:356] "Generic (PLEG): container finished" podID="6d89558b-7909-4d86-9305-720c63771d05" containerID="6e38faa53ef9860a61bfaf406e80af578c7f05723c01b146004605bf1c0847f9" exitCode=0 Oct 31 00:48:53.674859 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.674803 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" event={"ID":"6d89558b-7909-4d86-9305-720c63771d05","Type":"ContainerDied","Data":"6e38faa53ef9860a61bfaf406e80af578c7f05723c01b146004605bf1c0847f9"} Oct 31 00:48:53.677096 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.677068 4519 generic.go:356] "Generic (PLEG): container finished" podID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerID="562e889c23398bd92b8c294c6094af7e8ba054dd567fb57b0e7f6c22164de97e" exitCode=0 Oct 31 00:48:53.677241 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.677157 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" event={"ID":"55e0cdd8-4547-44da-b720-b2e35ee69846","Type":"ContainerDied","Data":"562e889c23398bd92b8c294c6094af7e8ba054dd567fb57b0e7f6c22164de97e"} Oct 31 00:48:53.678791 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.678771 4519 generic.go:356] "Generic (PLEG): container finished" podID="638ee841-3ada-4014-8841-f9da201ea000" containerID="7012a352ea43e9db0648a6f00bda576fe5333c4fd993ba9d93ad5783dbc6fe7d" exitCode=0 Oct 31 00:48:53.679002 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.678987 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" Oct 31 00:48:53.679399 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.679381 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" event={"ID":"638ee841-3ada-4014-8841-f9da201ea000","Type":"ContainerDied","Data":"7012a352ea43e9db0648a6f00bda576fe5333c4fd993ba9d93ad5783dbc6fe7d"} Oct 31 00:48:53.679491 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.679423 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph" event={"ID":"638ee841-3ada-4014-8841-f9da201ea000","Type":"ContainerDied","Data":"92993e7d62fc191a4de90ee0dd999a73aa199c0bc655d7bbbabc7e54ceecf03c"} Oct 31 00:48:53.679491 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.679477 4519 scope.go:122] "RemoveContainer" containerID="7012a352ea43e9db0648a6f00bda576fe5333c4fd993ba9d93ad5783dbc6fe7d" Oct 31 00:48:53.682370 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.682350 4519 generic.go:356] "Generic (PLEG): container finished" podID="06a6761a-4e9e-4675-8dec-dea23f865635" containerID="87d40eed09e0c1f606cb69ac3819cefb70bb9d9e4d5c6d3abbfe3267182ef87c" exitCode=0 Oct 31 00:48:53.682709 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.682401 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" event={"ID":"06a6761a-4e9e-4675-8dec-dea23f865635","Type":"ContainerDied","Data":"87d40eed09e0c1f606cb69ac3819cefb70bb9d9e4d5c6d3abbfe3267182ef87c"} Oct 31 00:48:53.696009 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.695986 4519 scope.go:122] "RemoveContainer" containerID="7012a352ea43e9db0648a6f00bda576fe5333c4fd993ba9d93ad5783dbc6fe7d" Oct 31 00:48:53.696339 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:48:53.696313 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7012a352ea43e9db0648a6f00bda576fe5333c4fd993ba9d93ad5783dbc6fe7d\": container with ID starting with 7012a352ea43e9db0648a6f00bda576fe5333c4fd993ba9d93ad5783dbc6fe7d not found: ID does not exist" containerID="7012a352ea43e9db0648a6f00bda576fe5333c4fd993ba9d93ad5783dbc6fe7d" Oct 31 00:48:53.696441 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.696351 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7012a352ea43e9db0648a6f00bda576fe5333c4fd993ba9d93ad5783dbc6fe7d"} err="failed to get container status \"7012a352ea43e9db0648a6f00bda576fe5333c4fd993ba9d93ad5783dbc6fe7d\": rpc error: code = NotFound desc = could not find container \"7012a352ea43e9db0648a6f00bda576fe5333c4fd993ba9d93ad5783dbc6fe7d\": container with ID starting with 7012a352ea43e9db0648a6f00bda576fe5333c4fd993ba9d93ad5783dbc6fe7d not found: ID does not exist" Oct 31 00:48:53.721480 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.721451 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:48:53.776881 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.776756 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-istio-data\") pod \"5498f31e-f243-43b8-992a-56cbccdc239c\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " Oct 31 00:48:53.776881 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.776803 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5498f31e-f243-43b8-992a-56cbccdc239c-kube-api-access-5clh4\" (UniqueName: \"kubernetes.io/projected/5498f31e-f243-43b8-992a-56cbccdc239c-kube-api-access-5clh4\") pod \"5498f31e-f243-43b8-992a-56cbccdc239c\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " Oct 31 00:48:53.776881 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.776832 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5498f31e-f243-43b8-992a-56cbccdc239c-istio-token\" (UniqueName: \"kubernetes.io/projected/5498f31e-f243-43b8-992a-56cbccdc239c-istio-token\") pod \"5498f31e-f243-43b8-992a-56cbccdc239c\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " Oct 31 00:48:53.777150 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.776899 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-credential-socket\") pod \"5498f31e-f243-43b8-992a-56cbccdc239c\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " Oct 31 00:48:53.777150 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.776980 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638ee841-3ada-4014-8841-f9da201ea000-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638ee841-3ada-4014-8841-f9da201ea000-maas-api-tls\") pod \"638ee841-3ada-4014-8841-f9da201ea000\" (UID: \"638ee841-3ada-4014-8841-f9da201ea000\") " Oct 31 00:48:53.777150 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.777042 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5498f31e-f243-43b8-992a-56cbccdc239c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5498f31e-f243-43b8-992a-56cbccdc239c-istio-podinfo\") pod \"5498f31e-f243-43b8-992a-56cbccdc239c\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " Oct 31 00:48:53.777150 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.777077 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-workload-certs\") pod \"5498f31e-f243-43b8-992a-56cbccdc239c\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " Oct 31 00:48:53.777274 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.777164 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-workload-socket\") pod \"5498f31e-f243-43b8-992a-56cbccdc239c\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " Oct 31 00:48:53.777274 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.777193 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-istio-envoy\") pod \"5498f31e-f243-43b8-992a-56cbccdc239c\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " Oct 31 00:48:53.777274 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.777236 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5498f31e-f243-43b8-992a-56cbccdc239c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5498f31e-f243-43b8-992a-56cbccdc239c-istiod-ca-cert\") pod \"5498f31e-f243-43b8-992a-56cbccdc239c\" (UID: \"5498f31e-f243-43b8-992a-56cbccdc239c\") " Oct 31 00:48:53.777371 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.777271 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638ee841-3ada-4014-8841-f9da201ea000-kube-api-access-vf5v2\" (UniqueName: \"kubernetes.io/projected/638ee841-3ada-4014-8841-f9da201ea000-kube-api-access-vf5v2\") pod \"638ee841-3ada-4014-8841-f9da201ea000\" (UID: \"638ee841-3ada-4014-8841-f9da201ea000\") " Oct 31 00:48:53.777371 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.777301 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638ee841-3ada-4014-8841-f9da201ea000-metrics-certs\" (UniqueName: \"kubernetes.io/secret/638ee841-3ada-4014-8841-f9da201ea000-metrics-certs\") pod \"638ee841-3ada-4014-8841-f9da201ea000\" (UID: \"638ee841-3ada-4014-8841-f9da201ea000\") " Oct 31 00:48:53.777371 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.777356 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/638ee841-3ada-4014-8841-f9da201ea000-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638ee841-3ada-4014-8841-f9da201ea000-openshift-service-ca\") pod \"638ee841-3ada-4014-8841-f9da201ea000\" (UID: \"638ee841-3ada-4014-8841-f9da201ea000\") " Oct 31 00:48:53.779179 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.779147 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-istio-data" pod "5498f31e-f243-43b8-992a-56cbccdc239c" (UID: "5498f31e-f243-43b8-992a-56cbccdc239c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.779393 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.779375 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-workload-socket" pod "5498f31e-f243-43b8-992a-56cbccdc239c" (UID: "5498f31e-f243-43b8-992a-56cbccdc239c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.779862 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.779837 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-workload-certs" pod "5498f31e-f243-43b8-992a-56cbccdc239c" (UID: "5498f31e-f243-43b8-992a-56cbccdc239c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.780628 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.780511 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638ee841-3ada-4014-8841-f9da201ea000-openshift-service-ca" pod "638ee841-3ada-4014-8841-f9da201ea000" (UID: "638ee841-3ada-4014-8841-f9da201ea000"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:53.782597 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.782459 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-credential-socket" pod "5498f31e-f243-43b8-992a-56cbccdc239c" (UID: "5498f31e-f243-43b8-992a-56cbccdc239c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.783977 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.783821 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638ee841-3ada-4014-8841-f9da201ea000-kube-api-access-vf5v2" pod "638ee841-3ada-4014-8841-f9da201ea000" (UID: "638ee841-3ada-4014-8841-f9da201ea000"). InnerVolumeSpecName "kube-api-access-vf5v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:53.784345 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.784322 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5498f31e-f243-43b8-992a-56cbccdc239c-istiod-ca-cert" pod "5498f31e-f243-43b8-992a-56cbccdc239c" (UID: "5498f31e-f243-43b8-992a-56cbccdc239c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:53.786902 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.786143 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5498f31e-f243-43b8-992a-56cbccdc239c-kube-api-access-5clh4" pod "5498f31e-f243-43b8-992a-56cbccdc239c" (UID: "5498f31e-f243-43b8-992a-56cbccdc239c"). InnerVolumeSpecName "kube-api-access-5clh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:53.786902 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.786230 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5498f31e-f243-43b8-992a-56cbccdc239c-istio-podinfo" pod "5498f31e-f243-43b8-992a-56cbccdc239c" (UID: "5498f31e-f243-43b8-992a-56cbccdc239c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:48:53.786902 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.786319 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-istio-envoy" pod "5498f31e-f243-43b8-992a-56cbccdc239c" (UID: "5498f31e-f243-43b8-992a-56cbccdc239c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.786902 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.786555 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638ee841-3ada-4014-8841-f9da201ea000-maas-api-tls" pod "638ee841-3ada-4014-8841-f9da201ea000" (UID: "638ee841-3ada-4014-8841-f9da201ea000"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:53.786902 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.786585 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5498f31e-f243-43b8-992a-56cbccdc239c-istio-token" pod "5498f31e-f243-43b8-992a-56cbccdc239c" (UID: "5498f31e-f243-43b8-992a-56cbccdc239c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:53.791758 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.788215 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638ee841-3ada-4014-8841-f9da201ea000-metrics-certs" pod "638ee841-3ada-4014-8841-f9da201ea000" (UID: "638ee841-3ada-4014-8841-f9da201ea000"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:53.841738 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.841708 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:48:53.851571 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.851545 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:48:53.873559 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.873523 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:53.878545 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.878516 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e0cdd8-4547-44da-b720-b2e35ee69846-kube-api-access-7c9v6\" (UniqueName: \"kubernetes.io/projected/55e0cdd8-4547-44da-b720-b2e35ee69846-kube-api-access-7c9v6\") pod \"55e0cdd8-4547-44da-b720-b2e35ee69846\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " Oct 31 00:48:53.878739 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.878581 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-workload-socket\") pod \"55e0cdd8-4547-44da-b720-b2e35ee69846\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " Oct 31 00:48:53.878739 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.878626 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e0cdd8-4547-44da-b720-b2e35ee69846-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e0cdd8-4547-44da-b720-b2e35ee69846-istiod-ca-cert\") pod \"55e0cdd8-4547-44da-b720-b2e35ee69846\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " Oct 31 00:48:53.878739 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.878717 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-workload-certs\") pod \"6d89558b-7909-4d86-9305-720c63771d05\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " Oct 31 00:48:53.878992 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.878746 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-istio-envoy\") pod \"6d89558b-7909-4d86-9305-720c63771d05\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " Oct 31 00:48:53.878992 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.878787 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-istio-data\") pod \"6d89558b-7909-4d86-9305-720c63771d05\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " Oct 31 00:48:53.878992 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.878822 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-data\") pod \"55e0cdd8-4547-44da-b720-b2e35ee69846\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " Oct 31 00:48:53.878992 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.878853 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-workload-socket\") pod \"6d89558b-7909-4d86-9305-720c63771d05\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " Oct 31 00:48:53.878992 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.878883 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-token\" (UniqueName: \"kubernetes.io/projected/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-token\") pod \"55e0cdd8-4547-44da-b720-b2e35ee69846\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " Oct 31 00:48:53.878992 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.878922 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-podinfo\") pod \"55e0cdd8-4547-44da-b720-b2e35ee69846\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " Oct 31 00:48:53.878992 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.878967 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-credential-socket\") pod \"6d89558b-7909-4d86-9305-720c63771d05\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " Oct 31 00:48:53.878992 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.878995 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d89558b-7909-4d86-9305-720c63771d05-kube-api-access-xc2dn\" (UniqueName: \"kubernetes.io/projected/6d89558b-7909-4d86-9305-720c63771d05-kube-api-access-xc2dn\") pod \"6d89558b-7909-4d86-9305-720c63771d05\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " Oct 31 00:48:53.879414 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.878992 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-workload-socket" pod "55e0cdd8-4547-44da-b720-b2e35ee69846" (UID: "55e0cdd8-4547-44da-b720-b2e35ee69846"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.879414 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.879044 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-credential-socket\") pod \"55e0cdd8-4547-44da-b720-b2e35ee69846\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " Oct 31 00:48:53.879414 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.879100 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d89558b-7909-4d86-9305-720c63771d05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d89558b-7909-4d86-9305-720c63771d05-istiod-ca-cert\") pod \"6d89558b-7909-4d86-9305-720c63771d05\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " Oct 31 00:48:53.879414 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.879126 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-envoy\") pod \"55e0cdd8-4547-44da-b720-b2e35ee69846\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " Oct 31 00:48:53.879414 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.879146 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-istio-data" pod "6d89558b-7909-4d86-9305-720c63771d05" (UID: "6d89558b-7909-4d86-9305-720c63771d05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.879414 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.879155 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d89558b-7909-4d86-9305-720c63771d05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d89558b-7909-4d86-9305-720c63771d05-istio-podinfo\") pod \"6d89558b-7909-4d86-9305-720c63771d05\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " Oct 31 00:48:53.879414 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.879255 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-workload-certs\") pod \"55e0cdd8-4547-44da-b720-b2e35ee69846\" (UID: \"55e0cdd8-4547-44da-b720-b2e35ee69846\") " Oct 31 00:48:53.879414 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.879288 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d89558b-7909-4d86-9305-720c63771d05-istio-token\" (UniqueName: \"kubernetes.io/projected/6d89558b-7909-4d86-9305-720c63771d05-istio-token\") pod \"6d89558b-7909-4d86-9305-720c63771d05\" (UID: \"6d89558b-7909-4d86-9305-720c63771d05\") " Oct 31 00:48:53.880153 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.879152 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e0cdd8-4547-44da-b720-b2e35ee69846-istiod-ca-cert" pod "55e0cdd8-4547-44da-b720-b2e35ee69846" (UID: "55e0cdd8-4547-44da-b720-b2e35ee69846"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:53.880153 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.879898 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-workload-socket" pod "6d89558b-7909-4d86-9305-720c63771d05" (UID: "6d89558b-7909-4d86-9305-720c63771d05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.880153 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.879869 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-credential-socket" pod "55e0cdd8-4547-44da-b720-b2e35ee69846" (UID: "55e0cdd8-4547-44da-b720-b2e35ee69846"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.880153 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.879966 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-credential-socket" pod "6d89558b-7909-4d86-9305-720c63771d05" (UID: "6d89558b-7909-4d86-9305-720c63771d05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.880360 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880336 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-data" pod "55e0cdd8-4547-44da-b720-b2e35ee69846" (UID: "55e0cdd8-4547-44da-b720-b2e35ee69846"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.880546 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880522 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880552 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e0cdd8-4547-44da-b720-b2e35ee69846-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880568 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880581 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880595 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880610 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5498f31e-f243-43b8-992a-56cbccdc239c-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880623 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880649 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880665 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf5v2\" (UniqueName: \"kubernetes.io/projected/638ee841-3ada-4014-8841-f9da201ea000-kube-api-access-vf5v2\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880680 4519 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/638ee841-3ada-4014-8841-f9da201ea000-metrics-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880695 4519 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638ee841-3ada-4014-8841-f9da201ea000-openshift-service-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880709 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880724 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880755 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880799 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5clh4\" (UniqueName: \"kubernetes.io/projected/5498f31e-f243-43b8-992a-56cbccdc239c-kube-api-access-5clh4\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880835 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5498f31e-f243-43b8-992a-56cbccdc239c-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880857 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498f31e-f243-43b8-992a-56cbccdc239c-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880873 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638ee841-3ada-4014-8841-f9da201ea000-maas-api-tls\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880905 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5498f31e-f243-43b8-992a-56cbccdc239c-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.881079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.880929 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.882275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.881333 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-workload-certs" pod "55e0cdd8-4547-44da-b720-b2e35ee69846" (UID: "55e0cdd8-4547-44da-b720-b2e35ee69846"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.882275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.881604 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-istio-envoy" pod "6d89558b-7909-4d86-9305-720c63771d05" (UID: "6d89558b-7909-4d86-9305-720c63771d05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.882462 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.882431 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e0cdd8-4547-44da-b720-b2e35ee69846-kube-api-access-7c9v6" pod "55e0cdd8-4547-44da-b720-b2e35ee69846" (UID: "55e0cdd8-4547-44da-b720-b2e35ee69846"). InnerVolumeSpecName "kube-api-access-7c9v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:53.882840 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.882817 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d89558b-7909-4d86-9305-720c63771d05-istiod-ca-cert" pod "6d89558b-7909-4d86-9305-720c63771d05" (UID: "6d89558b-7909-4d86-9305-720c63771d05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:53.883042 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.883022 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-workload-certs" pod "6d89558b-7909-4d86-9305-720c63771d05" (UID: "6d89558b-7909-4d86-9305-720c63771d05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.887255 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.887230 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d89558b-7909-4d86-9305-720c63771d05-kube-api-access-xc2dn" pod "6d89558b-7909-4d86-9305-720c63771d05" (UID: "6d89558b-7909-4d86-9305-720c63771d05"). InnerVolumeSpecName "kube-api-access-xc2dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:53.887741 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.887706 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d89558b-7909-4d86-9305-720c63771d05-istio-token" pod "6d89558b-7909-4d86-9305-720c63771d05" (UID: "6d89558b-7909-4d86-9305-720c63771d05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:53.887828 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.887735 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d89558b-7909-4d86-9305-720c63771d05-istio-podinfo" pod "6d89558b-7909-4d86-9305-720c63771d05" (UID: "6d89558b-7909-4d86-9305-720c63771d05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:48:53.888420 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.888399 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-token" pod "55e0cdd8-4547-44da-b720-b2e35ee69846" (UID: "55e0cdd8-4547-44da-b720-b2e35ee69846"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:53.889399 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.889358 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-envoy" pod "55e0cdd8-4547-44da-b720-b2e35ee69846" (UID: "55e0cdd8-4547-44da-b720-b2e35ee69846"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.893938 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.893915 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-podinfo" pod "55e0cdd8-4547-44da-b720-b2e35ee69846" (UID: "55e0cdd8-4547-44da-b720-b2e35ee69846"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:48:53.896625 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.896604 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:48:53.981344 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.981309 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-istio-data\") pod \"06a6761a-4e9e-4675-8dec-dea23f865635\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " Oct 31 00:48:53.981344 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.981351 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-istio-envoy\") pod \"06a6761a-4e9e-4675-8dec-dea23f865635\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " Oct 31 00:48:53.982052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.981398 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a6761a-4e9e-4675-8dec-dea23f865635-kube-api-access-b5hz9\" (UniqueName: \"kubernetes.io/projected/06a6761a-4e9e-4675-8dec-dea23f865635-kube-api-access-b5hz9\") pod \"06a6761a-4e9e-4675-8dec-dea23f865635\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " Oct 31 00:48:53.982052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.981467 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06a6761a-4e9e-4675-8dec-dea23f865635-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06a6761a-4e9e-4675-8dec-dea23f865635-istio-podinfo\") pod \"06a6761a-4e9e-4675-8dec-dea23f865635\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " Oct 31 00:48:53.982052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.981596 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a6761a-4e9e-4675-8dec-dea23f865635-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06a6761a-4e9e-4675-8dec-dea23f865635-istiod-ca-cert\") pod \"06a6761a-4e9e-4675-8dec-dea23f865635\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " Oct 31 00:48:53.982052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.981610 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-istio-data" pod "06a6761a-4e9e-4675-8dec-dea23f865635" (UID: "06a6761a-4e9e-4675-8dec-dea23f865635"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.982052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.981672 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-workload-socket\") pod \"06a6761a-4e9e-4675-8dec-dea23f865635\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " Oct 31 00:48:53.982052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.981703 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a6761a-4e9e-4675-8dec-dea23f865635-istio-token\" (UniqueName: \"kubernetes.io/projected/06a6761a-4e9e-4675-8dec-dea23f865635-istio-token\") pod \"06a6761a-4e9e-4675-8dec-dea23f865635\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " Oct 31 00:48:53.982052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.981736 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-workload-certs\") pod \"06a6761a-4e9e-4675-8dec-dea23f865635\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " Oct 31 00:48:53.982052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.981763 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-credential-socket\") pod \"06a6761a-4e9e-4675-8dec-dea23f865635\" (UID: \"06a6761a-4e9e-4675-8dec-dea23f865635\") " Oct 31 00:48:53.982052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.981955 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-workload-socket" pod "06a6761a-4e9e-4675-8dec-dea23f865635" (UID: "06a6761a-4e9e-4675-8dec-dea23f865635"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.982052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.981975 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a6761a-4e9e-4675-8dec-dea23f865635-istiod-ca-cert" pod "06a6761a-4e9e-4675-8dec-dea23f865635" (UID: "06a6761a-4e9e-4675-8dec-dea23f865635"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:53.982481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982179 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-workload-certs" pod "06a6761a-4e9e-4675-8dec-dea23f865635" (UID: "06a6761a-4e9e-4675-8dec-dea23f865635"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.982481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982194 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-credential-socket" pod "06a6761a-4e9e-4675-8dec-dea23f865635" (UID: "06a6761a-4e9e-4675-8dec-dea23f865635"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.982481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982367 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.982481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982387 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.982481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982403 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc2dn\" (UniqueName: \"kubernetes.io/projected/6d89558b-7909-4d86-9305-720c63771d05-kube-api-access-xc2dn\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.982481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982418 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06a6761a-4e9e-4675-8dec-dea23f865635-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.982481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982433 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d89558b-7909-4d86-9305-720c63771d05-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.982481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982447 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.982481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982460 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.982481 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982477 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d89558b-7909-4d86-9305-720c63771d05-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.982825 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982491 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.982825 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982504 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.982825 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982518 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e0cdd8-4547-44da-b720-b2e35ee69846-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.982825 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982530 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.982825 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982568 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d89558b-7909-4d86-9305-720c63771d05-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.982825 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982583 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c9v6\" (UniqueName: \"kubernetes.io/projected/55e0cdd8-4547-44da-b720-b2e35ee69846-kube-api-access-7c9v6\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.982825 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982598 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.982825 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.982612 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d89558b-7909-4d86-9305-720c63771d05-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:53.984048 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.984020 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-istio-envoy" pod "06a6761a-4e9e-4675-8dec-dea23f865635" (UID: "06a6761a-4e9e-4675-8dec-dea23f865635"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:53.984235 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.984211 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a6761a-4e9e-4675-8dec-dea23f865635-kube-api-access-b5hz9" pod "06a6761a-4e9e-4675-8dec-dea23f865635" (UID: "06a6761a-4e9e-4675-8dec-dea23f865635"). InnerVolumeSpecName "kube-api-access-b5hz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:53.984348 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.984330 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a6761a-4e9e-4675-8dec-dea23f865635-istio-token" pod "06a6761a-4e9e-4675-8dec-dea23f865635" (UID: "06a6761a-4e9e-4675-8dec-dea23f865635"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:53.984702 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:53.984677 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06a6761a-4e9e-4675-8dec-dea23f865635-istio-podinfo" pod "06a6761a-4e9e-4675-8dec-dea23f865635" (UID: "06a6761a-4e9e-4675-8dec-dea23f865635"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:48:54.016444 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.016408 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph"] Oct 31 00:48:54.027791 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.027759 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f99ac0-75d498cd5c-hqxph"] Oct 31 00:48:54.084348 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.084236 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06a6761a-4e9e-4675-8dec-dea23f865635-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:54.084348 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.084282 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5hz9\" (UniqueName: \"kubernetes.io/projected/06a6761a-4e9e-4675-8dec-dea23f865635-kube-api-access-b5hz9\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:54.084348 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.084303 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06a6761a-4e9e-4675-8dec-dea23f865635-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:54.084348 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.084320 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06a6761a-4e9e-4675-8dec-dea23f865635-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:54.102360 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.102333 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="638ee841-3ada-4014-8841-f9da201ea000" path="/var/lib/kubelet/pods/638ee841-3ada-4014-8841-f9da201ea000/volumes" Oct 31 00:48:54.102788 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.102773 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ece35843-f7a2-481a-b854-8b15bfdbb38f" path="/var/lib/kubelet/pods/ece35843-f7a2-481a-b854-8b15bfdbb38f/volumes" Oct 31 00:48:54.496436 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.496400 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:54.500376 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.500348 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" podUID="f76bbc2e-d9b0-4ac4-a09d-91b609760c16" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:48:54.690431 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.690399 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" Oct 31 00:48:54.690605 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.690397 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh" event={"ID":"5498f31e-f243-43b8-992a-56cbccdc239c","Type":"ContainerDied","Data":"c5ca46aa5ac983bbcc7096115815db67ebdec49fa41010e55172d0bb9537289f"} Oct 31 00:48:54.690605 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.690538 4519 scope.go:122] "RemoveContainer" containerID="bd61ba53498b29b792c2194cdbdd2024201a24521590e2e096d86e3479cf1cb9" Oct 31 00:48:54.692039 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.692021 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" Oct 31 00:48:54.692450 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.692429 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr" event={"ID":"6d89558b-7909-4d86-9305-720c63771d05","Type":"ContainerDied","Data":"02d7e4fcdd995cb9c0c352fb01c884bea87c1039fdf48e84493c707133ceccf5"} Oct 31 00:48:54.694281 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.694256 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" event={"ID":"55e0cdd8-4547-44da-b720-b2e35ee69846","Type":"ContainerDied","Data":"7cf1674a31414bce3397435d220879ac6db2bd982f86a9725dc3221985fd97ae"} Oct 31 00:48:54.694410 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.694264 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w" Oct 31 00:48:54.697267 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.697247 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" event={"ID":"06a6761a-4e9e-4675-8dec-dea23f865635","Type":"ContainerDied","Data":"93fa8053d344c13f516387883f62949ec6ce9589617800b83f165f5dea769ee9"} Oct 31 00:48:54.697407 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.697275 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4" Oct 31 00:48:54.702361 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.702335 4519 scope.go:122] "RemoveContainer" containerID="6e38faa53ef9860a61bfaf406e80af578c7f05723c01b146004605bf1c0847f9" Oct 31 00:48:54.716364 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.716339 4519 scope.go:122] "RemoveContainer" containerID="562e889c23398bd92b8c294c6094af7e8ba054dd567fb57b0e7f6c22164de97e" Oct 31 00:48:54.728390 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.728330 4519 scope.go:122] "RemoveContainer" containerID="87d40eed09e0c1f606cb69ac3819cefb70bb9d9e4d5c6d3abbfe3267182ef87c" Oct 31 00:48:54.734940 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.734916 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh"] Oct 31 00:48:54.743418 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.743394 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:54.751279 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.751221 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-tz7hh"] Oct 31 00:48:54.767480 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.767454 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w"] Oct 31 00:48:54.780578 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.780551 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-sjs4w"] Oct 31 00:48:54.817242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.817208 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr"] Oct 31 00:48:54.822014 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.821986 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-wvgqr"] Oct 31 00:48:54.835659 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.835604 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:54.838098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.838079 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4"] Oct 31 00:48:54.846884 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:54.846858 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f1496-openshift-default-8665bf5969-cb7j4"] Oct 31 00:48:55.574771 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:55.574736 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:56.109326 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.109285 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" path="/var/lib/kubelet/pods/06a6761a-4e9e-4675-8dec-dea23f865635/volumes" Oct 31 00:48:56.109934 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.109918 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5498f31e-f243-43b8-992a-56cbccdc239c" path="/var/lib/kubelet/pods/5498f31e-f243-43b8-992a-56cbccdc239c/volumes" Oct 31 00:48:56.110473 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.110459 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" path="/var/lib/kubelet/pods/55e0cdd8-4547-44da-b720-b2e35ee69846/volumes" Oct 31 00:48:56.111477 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.111460 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d89558b-7909-4d86-9305-720c63771d05" path="/var/lib/kubelet/pods/6d89558b-7909-4d86-9305-720c63771d05/volumes" Oct 31 00:48:56.134629 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.134595 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8"] Oct 31 00:48:56.135052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135038 4519 cpu_manager.go:414] "containerMap: removing container" podUID="f6b39483-2371-4570-843b-c4421cad360f" containerName="istio-proxy" Oct 31 00:48:56.135095 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135053 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b39483-2371-4570-843b-c4421cad360f" containerName="istio-proxy" Oct 31 00:48:56.135095 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135065 4519 cpu_manager.go:414] "containerMap: removing container" podUID="5498f31e-f243-43b8-992a-56cbccdc239c" containerName="istio-proxy" Oct 31 00:48:56.135095 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135071 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5498f31e-f243-43b8-992a-56cbccdc239c" containerName="istio-proxy" Oct 31 00:48:56.135095 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135079 4519 cpu_manager.go:414] "containerMap: removing container" podUID="4320a978-89eb-4601-a10e-028e2549b7dd" containerName="istio-proxy" Oct 31 00:48:56.135095 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135084 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4320a978-89eb-4601-a10e-028e2549b7dd" containerName="istio-proxy" Oct 31 00:48:56.135095 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135090 4519 cpu_manager.go:414] "containerMap: removing container" podUID="393b6b08-821f-4788-9da8-230f14d4e867" containerName="istio-proxy" Oct 31 00:48:56.135095 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135094 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393b6b08-821f-4788-9da8-230f14d4e867" containerName="istio-proxy" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135103 4519 cpu_manager.go:414] "containerMap: removing container" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135107 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135115 4519 cpu_manager.go:414] "containerMap: removing container" podUID="a10862b6-342b-4eb3-9c1a-5ed66b45ba02" containerName="revoke-keys" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135119 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10862b6-342b-4eb3-9c1a-5ed66b45ba02" containerName="revoke-keys" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135126 4519 cpu_manager.go:414] "containerMap: removing container" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135130 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135148 4519 cpu_manager.go:414] "containerMap: removing container" podUID="638ee841-3ada-4014-8841-f9da201ea000" containerName="maas-api" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135152 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638ee841-3ada-4014-8841-f9da201ea000" containerName="maas-api" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135160 4519 cpu_manager.go:414] "containerMap: removing container" podUID="2ec76a17-5431-4756-a824-862c354595c8" containerName="istio-proxy" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135164 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec76a17-5431-4756-a824-862c354595c8" containerName="istio-proxy" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135169 4519 cpu_manager.go:414] "containerMap: removing container" podUID="6d89558b-7909-4d86-9305-720c63771d05" containerName="istio-proxy" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135173 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d89558b-7909-4d86-9305-720c63771d05" containerName="istio-proxy" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135183 4519 cpu_manager.go:414] "containerMap: removing container" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135188 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135243 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec76a17-5431-4756-a824-862c354595c8" containerName="istio-proxy" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135249 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d89558b-7909-4d86-9305-720c63771d05" containerName="istio-proxy" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135255 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27be156-a2bf-407a-a15c-b348f3e87105" containerName="main" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135261 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b39483-2371-4570-843b-c4421cad360f" containerName="istio-proxy" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135268 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5498f31e-f243-43b8-992a-56cbccdc239c" containerName="istio-proxy" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135273 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4320a978-89eb-4601-a10e-028e2549b7dd" containerName="istio-proxy" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135278 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="393b6b08-821f-4788-9da8-230f14d4e867" containerName="istio-proxy" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135284 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e0cdd8-4547-44da-b720-b2e35ee69846" containerName="istio-proxy" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135289 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10862b6-342b-4eb3-9c1a-5ed66b45ba02" containerName="revoke-keys" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135295 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a6761a-4e9e-4675-8dec-dea23f865635" containerName="istio-proxy" Oct 31 00:48:56.135300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.135302 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="638ee841-3ada-4014-8841-f9da201ea000" containerName="maas-api" Oct 31 00:48:56.140145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.140119 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8" Oct 31 00:48:56.156672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.147743 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8"] Oct 31 00:48:56.212414 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.212380 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/234959cb-ee8c-4761-9032-ec19e95d9106-maas-api-xks-ca\") pod \"maas-api-revoke-sub-742129c9-lvtv8\" (UID: \"234959cb-ee8c-4761-9032-ec19e95d9106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8" Oct 31 00:48:56.212624 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.212585 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/234959cb-ee8c-4761-9032-ec19e95d9106-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-742129c9-lvtv8\" (UID: \"234959cb-ee8c-4761-9032-ec19e95d9106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8" Oct 31 00:48:56.212715 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.212624 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsrwv\" (UniqueName: \"kubernetes.io/projected/234959cb-ee8c-4761-9032-ec19e95d9106-kube-api-access-fsrwv\") pod \"maas-api-revoke-sub-742129c9-lvtv8\" (UID: \"234959cb-ee8c-4761-9032-ec19e95d9106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8" Oct 31 00:48:56.239583 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.239550 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:56.239762 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.239610 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:48:56.314253 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.314221 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/234959cb-ee8c-4761-9032-ec19e95d9106-maas-api-xks-ca\") pod \"maas-api-revoke-sub-742129c9-lvtv8\" (UID: \"234959cb-ee8c-4761-9032-ec19e95d9106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8" Oct 31 00:48:56.314464 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.314384 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/234959cb-ee8c-4761-9032-ec19e95d9106-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-742129c9-lvtv8\" (UID: \"234959cb-ee8c-4761-9032-ec19e95d9106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8" Oct 31 00:48:56.314464 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.314419 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrwv\" (UniqueName: \"kubernetes.io/projected/234959cb-ee8c-4761-9032-ec19e95d9106-kube-api-access-fsrwv\") pod \"maas-api-revoke-sub-742129c9-lvtv8\" (UID: \"234959cb-ee8c-4761-9032-ec19e95d9106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8" Oct 31 00:48:56.315031 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.315010 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/234959cb-ee8c-4761-9032-ec19e95d9106-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-742129c9-lvtv8\" (UID: \"234959cb-ee8c-4761-9032-ec19e95d9106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8" Oct 31 00:48:56.316579 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.316554 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/234959cb-ee8c-4761-9032-ec19e95d9106-maas-api-xks-ca\") pod \"maas-api-revoke-sub-742129c9-lvtv8\" (UID: \"234959cb-ee8c-4761-9032-ec19e95d9106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8" Oct 31 00:48:56.323001 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.322977 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsrwv\" (UniqueName: \"kubernetes.io/projected/234959cb-ee8c-4761-9032-ec19e95d9106-kube-api-access-fsrwv\") pod \"maas-api-revoke-sub-742129c9-lvtv8\" (UID: \"234959cb-ee8c-4761-9032-ec19e95d9106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8" Oct 31 00:48:56.459617 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.459537 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8" Oct 31 00:48:56.771099 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:56.771065 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8"] Oct 31 00:48:56.771530 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:48:56.771461 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234959cb_ee8c_4761_9032_ec19e95d9106.slice/crio-745eaa959e89ebc8e998b04a34e72bd6a4d58c42ec3371313e400d9f5e0361dd WatchSource:0}: Error finding container 745eaa959e89ebc8e998b04a34e72bd6a4d58c42ec3371313e400d9f5e0361dd: Status 404 returned error can't find the container with id 745eaa959e89ebc8e998b04a34e72bd6a4d58c42ec3371313e400d9f5e0361dd Oct 31 00:48:57.684096 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:57.684028 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" containerID="cri-o://3c3bd621840c39b9b4513d91a54f8fb09e4d437e54e27be3e2e4966c3cb5c2ac" gracePeriod=45 Oct 31 00:48:57.727079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:57.727038 4519 generic.go:356] "Generic (PLEG): container finished" podID="234959cb-ee8c-4761-9032-ec19e95d9106" containerID="2edd1c505c031de3290b4febe911e89cbee0c8ca4605b95f6fd30ef7f4253e34" exitCode=0 Oct 31 00:48:57.727247 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:57.727134 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8" event={"ID":"234959cb-ee8c-4761-9032-ec19e95d9106","Type":"ContainerDied","Data":"2edd1c505c031de3290b4febe911e89cbee0c8ca4605b95f6fd30ef7f4253e34"} Oct 31 00:48:57.727247 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:57.727188 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8" event={"ID":"234959cb-ee8c-4761-9032-ec19e95d9106","Type":"ContainerStarted","Data":"745eaa959e89ebc8e998b04a34e72bd6a4d58c42ec3371313e400d9f5e0361dd"} Oct 31 00:48:57.782073 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:57.781978 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": dial tcp 10.133.0.44:8000: connect: connection refused" Oct 31 00:48:57.943998 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:57.943972 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:48:58.032548 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.032451 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-tmp-dir\") pod \"63751bb9-e9b0-4c48-9887-2bf875a39d87\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " Oct 31 00:48:58.032717 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.032571 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63751bb9-e9b0-4c48-9887-2bf875a39d87-tls-certs\" (UniqueName: \"kubernetes.io/secret/63751bb9-e9b0-4c48-9887-2bf875a39d87-tls-certs\") pod \"63751bb9-e9b0-4c48-9887-2bf875a39d87\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " Oct 31 00:48:58.032717 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.032621 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-model-cache\") pod \"63751bb9-e9b0-4c48-9887-2bf875a39d87\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " Oct 31 00:48:58.032717 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.032662 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-home\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-home\") pod \"63751bb9-e9b0-4c48-9887-2bf875a39d87\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " Oct 31 00:48:58.032884 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.032738 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-dshm\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-dshm\") pod \"63751bb9-e9b0-4c48-9887-2bf875a39d87\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " Oct 31 00:48:58.032884 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.032741 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-tmp-dir" pod "63751bb9-e9b0-4c48-9887-2bf875a39d87" (UID: "63751bb9-e9b0-4c48-9887-2bf875a39d87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:58.032884 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.032829 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63751bb9-e9b0-4c48-9887-2bf875a39d87-kube-api-access-cvgvl\" (UniqueName: \"kubernetes.io/projected/63751bb9-e9b0-4c48-9887-2bf875a39d87-kube-api-access-cvgvl\") pod \"63751bb9-e9b0-4c48-9887-2bf875a39d87\" (UID: \"63751bb9-e9b0-4c48-9887-2bf875a39d87\") " Oct 31 00:48:58.033048 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.032893 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-model-cache" pod "63751bb9-e9b0-4c48-9887-2bf875a39d87" (UID: "63751bb9-e9b0-4c48-9887-2bf875a39d87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:58.033048 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.032976 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-home" pod "63751bb9-e9b0-4c48-9887-2bf875a39d87" (UID: "63751bb9-e9b0-4c48-9887-2bf875a39d87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:58.033282 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.033260 4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-tmp-dir\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:58.033346 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.033288 4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-model-cache\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:58.033346 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.033303 4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-home\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:58.034894 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.034872 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63751bb9-e9b0-4c48-9887-2bf875a39d87-tls-certs" pod "63751bb9-e9b0-4c48-9887-2bf875a39d87" (UID: "63751bb9-e9b0-4c48-9887-2bf875a39d87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:58.034993 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.034913 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-dshm" pod "63751bb9-e9b0-4c48-9887-2bf875a39d87" (UID: "63751bb9-e9b0-4c48-9887-2bf875a39d87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:58.034993 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.034948 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63751bb9-e9b0-4c48-9887-2bf875a39d87-kube-api-access-cvgvl" pod "63751bb9-e9b0-4c48-9887-2bf875a39d87" (UID: "63751bb9-e9b0-4c48-9887-2bf875a39d87"). InnerVolumeSpecName "kube-api-access-cvgvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:58.135243 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.135205 4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63751bb9-e9b0-4c48-9887-2bf875a39d87-tls-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:58.135243 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.135246 4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63751bb9-e9b0-4c48-9887-2bf875a39d87-dshm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:58.135478 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.135262 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvgvl\" (UniqueName: \"kubernetes.io/projected/63751bb9-e9b0-4c48-9887-2bf875a39d87-kube-api-access-cvgvl\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:58.735112 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.735075 4519 generic.go:356] "Generic (PLEG): container finished" podID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerID="3c3bd621840c39b9b4513d91a54f8fb09e4d437e54e27be3e2e4966c3cb5c2ac" exitCode=0 Oct 31 00:48:58.735280 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.735120 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" Oct 31 00:48:58.735280 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.735165 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" event={"ID":"63751bb9-e9b0-4c48-9887-2bf875a39d87","Type":"ContainerDied","Data":"3c3bd621840c39b9b4513d91a54f8fb09e4d437e54e27be3e2e4966c3cb5c2ac"} Oct 31 00:48:58.735280 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.735209 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p" event={"ID":"63751bb9-e9b0-4c48-9887-2bf875a39d87","Type":"ContainerDied","Data":"8e020df3319688a765ac1763a258e851ae1123edb69cd9cc3e7e1d26a32824d0"} Oct 31 00:48:58.735280 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.735230 4519 scope.go:122] "RemoveContainer" containerID="3c3bd621840c39b9b4513d91a54f8fb09e4d437e54e27be3e2e4966c3cb5c2ac" Oct 31 00:48:58.745288 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.745264 4519 scope.go:122] "RemoveContainer" containerID="3c3bd621840c39b9b4513d91a54f8fb09e4d437e54e27be3e2e4966c3cb5c2ac" Oct 31 00:48:58.745603 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:48:58.745580 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c3bd621840c39b9b4513d91a54f8fb09e4d437e54e27be3e2e4966c3cb5c2ac\": container with ID starting with 3c3bd621840c39b9b4513d91a54f8fb09e4d437e54e27be3e2e4966c3cb5c2ac not found: ID does not exist" containerID="3c3bd621840c39b9b4513d91a54f8fb09e4d437e54e27be3e2e4966c3cb5c2ac" Oct 31 00:48:58.745696 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.745610 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c3bd621840c39b9b4513d91a54f8fb09e4d437e54e27be3e2e4966c3cb5c2ac"} err="failed to get container status \"3c3bd621840c39b9b4513d91a54f8fb09e4d437e54e27be3e2e4966c3cb5c2ac\": rpc error: code = NotFound desc = could not find container \"3c3bd621840c39b9b4513d91a54f8fb09e4d437e54e27be3e2e4966c3cb5c2ac\": container with ID starting with 3c3bd621840c39b9b4513d91a54f8fb09e4d437e54e27be3e2e4966c3cb5c2ac not found: ID does not exist" Oct 31 00:48:58.757371 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.757333 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p"] Oct 31 00:48:58.761326 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.761297 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-5c2a03/premium-w4-5c2a03-kserve-55bffd8c7d-gdm5p"] Oct 31 00:48:58.898362 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.898334 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8" Oct 31 00:48:58.944398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.944144 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/234959cb-ee8c-4761-9032-ec19e95d9106-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/234959cb-ee8c-4761-9032-ec19e95d9106-maas-api-xks-ca\") pod \"234959cb-ee8c-4761-9032-ec19e95d9106\" (UID: \"234959cb-ee8c-4761-9032-ec19e95d9106\") " Oct 31 00:48:58.944398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.944293 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/234959cb-ee8c-4761-9032-ec19e95d9106-kube-api-access-fsrwv\" (UniqueName: \"kubernetes.io/projected/234959cb-ee8c-4761-9032-ec19e95d9106-kube-api-access-fsrwv\") pod \"234959cb-ee8c-4761-9032-ec19e95d9106\" (UID: \"234959cb-ee8c-4761-9032-ec19e95d9106\") " Oct 31 00:48:58.944398 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.944371 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/234959cb-ee8c-4761-9032-ec19e95d9106-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/234959cb-ee8c-4761-9032-ec19e95d9106-maas-api-openshift-ca\") pod \"234959cb-ee8c-4761-9032-ec19e95d9106\" (UID: \"234959cb-ee8c-4761-9032-ec19e95d9106\") " Oct 31 00:48:58.945150 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.945124 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234959cb-ee8c-4761-9032-ec19e95d9106-maas-api-openshift-ca" pod "234959cb-ee8c-4761-9032-ec19e95d9106" (UID: "234959cb-ee8c-4761-9032-ec19e95d9106"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:58.947385 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.947345 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234959cb-ee8c-4761-9032-ec19e95d9106-maas-api-xks-ca" pod "234959cb-ee8c-4761-9032-ec19e95d9106" (UID: "234959cb-ee8c-4761-9032-ec19e95d9106"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:58.947498 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:58.947452 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234959cb-ee8c-4761-9032-ec19e95d9106-kube-api-access-fsrwv" pod "234959cb-ee8c-4761-9032-ec19e95d9106" (UID: "234959cb-ee8c-4761-9032-ec19e95d9106"). InnerVolumeSpecName "kube-api-access-fsrwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:59.046393 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:59.046311 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/234959cb-ee8c-4761-9032-ec19e95d9106-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:59.046393 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:59.046344 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/234959cb-ee8c-4761-9032-ec19e95d9106-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:59.046393 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:59.046355 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsrwv\" (UniqueName: \"kubernetes.io/projected/234959cb-ee8c-4761-9032-ec19e95d9106-kube-api-access-fsrwv\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:59.743195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:59.743160 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8" Oct 31 00:48:59.743839 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:59.743166 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8" event={"ID":"234959cb-ee8c-4761-9032-ec19e95d9106","Type":"ContainerDied","Data":"745eaa959e89ebc8e998b04a34e72bd6a4d58c42ec3371313e400d9f5e0361dd"} Oct 31 00:48:59.743839 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:59.743309 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="745eaa959e89ebc8e998b04a34e72bd6a4d58c42ec3371313e400d9f5e0361dd" Oct 31 00:48:59.981847 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:48:59.981815 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:00.096440 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:00.092773 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:00.103649 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:00.103604 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" path="/var/lib/kubelet/pods/63751bb9-e9b0-4c48-9887-2bf875a39d87/volumes" Oct 31 00:49:00.520243 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:00.520196 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq"] Oct 31 00:49:00.521389 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:00.521356 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" podUID="4edab197-6a3f-457a-acab-82d4cebc7b0f" containerName="istio-proxy" containerID="cri-o://6348482c530ea8af1d77c1fa72d36d3c3a545c2f1d3c44960310b79809c95142" gracePeriod=30 Oct 31 00:49:00.544721 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:00.538570 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv"] Oct 31 00:49:00.544721 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:00.538835 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" podUID="cdbf7b30-65d4-4462-af10-05cdd5015ce3" containerName="istio-proxy" containerID="cri-o://727939f6d98ac00afaa2991e4b057de2077f56d8f187b71aabc26ac40a1d97fb" gracePeriod=30 Oct 31 00:49:00.551910 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:00.551879 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w"] Oct 31 00:49:00.552294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:00.552275 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:00.766238 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:00.765190 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" containerID="cri-o://6fa1b78056d929bb107371a158e3918f3ef5908eb02a375be0e1f3a5935d4daa" gracePeriod=30 Oct 31 00:49:01.260830 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:01.260795 4519 patch_prober.go:29] interesting pod/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:49:01.261226 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:01.260844 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:49:03.532660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:03.532377 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" podUID="6b03043a-75b4-404a-800f-d71f33a04108" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:49:04.141765 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.141730 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" containerID="cri-o://73ef214360d8f16a916d2226135eb5455b1d62b7336fc626ee6ae391624265bb" gracePeriod=45 Oct 31 00:49:04.155261 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.155223 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" containerID="cri-o://9c2b9907c8b36a44e201bf4ecc634088534a7d3564c31aad87df5fd19e7152d7" gracePeriod=45 Oct 31 00:49:04.161557 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.161507 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" containerID="cri-o://1db6086e867eafebe67d526fded02d3390ff41791870007fb666a2e7f5e84546" gracePeriod=45 Oct 31 00:49:04.362816 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.354934 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:04.499848 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.497238 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:04.502029 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.501996 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" podUID="f76bbc2e-d9b0-4ac4-a09d-91b609760c16" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:49:04.508074 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.508047 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:49:04.510086 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.510060 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:49:04.540858 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.540823 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:49:04.576197 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.576164 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-home\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-home\") pod \"0d9d315d-8875-4e45-8d34-2005ddd628d5\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " Oct 31 00:49:04.576479 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.576459 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-dshm\") pod \"4298cde5-e51e-42f5-924d-0c2224a1f57e\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " Oct 31 00:49:04.576991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.576968 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d9d315d-8875-4e45-8d34-2005ddd628d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d9d315d-8875-4e45-8d34-2005ddd628d5-tls-certs\") pod \"0d9d315d-8875-4e45-8d34-2005ddd628d5\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " Oct 31 00:49:04.577143 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.577129 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-tls-certs\" (UniqueName: \"kubernetes.io/secret/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-tls-certs\") pod \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " Oct 31 00:49:04.577255 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.576583 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-home" pod "0d9d315d-8875-4e45-8d34-2005ddd628d5" (UID: "0d9d315d-8875-4e45-8d34-2005ddd628d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.577316 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d9d315d-8875-4e45-8d34-2005ddd628d5-kube-api-access-62mh5\" (UniqueName: \"kubernetes.io/projected/0d9d315d-8875-4e45-8d34-2005ddd628d5-kube-api-access-62mh5\") pod \"0d9d315d-8875-4e45-8d34-2005ddd628d5\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.577366 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-dshm\") pod \"0d9d315d-8875-4e45-8d34-2005ddd628d5\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.577397 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4298cde5-e51e-42f5-924d-0c2224a1f57e-tls-certs\" (UniqueName: \"kubernetes.io/secret/4298cde5-e51e-42f5-924d-0c2224a1f57e-tls-certs\") pod \"4298cde5-e51e-42f5-924d-0c2224a1f57e\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.577424 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-model-cache\") pod \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.577463 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-tmp-dir\") pod \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.577494 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-home\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-home\") pod \"4298cde5-e51e-42f5-924d-0c2224a1f57e\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.577529 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-model-cache\") pod \"4298cde5-e51e-42f5-924d-0c2224a1f57e\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.577569 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-tmp-dir\") pod \"0d9d315d-8875-4e45-8d34-2005ddd628d5\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.577601 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-home\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-home\") pod \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.577630 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-dshm\") pod \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.577904 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-tmp-dir" pod "193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" (UID: "193c8acd-7eb4-49f3-b2b3-0c398cdd9c02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.578099 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-model-cache" pod "193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" (UID: "193c8acd-7eb4-49f3-b2b3-0c398cdd9c02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.578298 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-model-cache" pod "4298cde5-e51e-42f5-924d-0c2224a1f57e" (UID: "4298cde5-e51e-42f5-924d-0c2224a1f57e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.578478 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-home" pod "4298cde5-e51e-42f5-924d-0c2224a1f57e" (UID: "4298cde5-e51e-42f5-924d-0c2224a1f57e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.578499 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-tmp-dir" pod "0d9d315d-8875-4e45-8d34-2005ddd628d5" (UID: "0d9d315d-8875-4e45-8d34-2005ddd628d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.578679 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-home" pod "193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" (UID: "193c8acd-7eb4-49f3-b2b3-0c398cdd9c02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.578922 4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-home\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.578947 4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-model-cache\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.578963 4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-tmp-dir\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.578976 4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-home\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.578989 4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-model-cache\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.579002 4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-tmp-dir\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.579041 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.579015 4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-home\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.587417 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.587383 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:04.591052 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.591022 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-dshm" pod "193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" (UID: "193c8acd-7eb4-49f3-b2b3-0c398cdd9c02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:04.591208 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.591160 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4298cde5-e51e-42f5-924d-0c2224a1f57e-tls-certs" pod "4298cde5-e51e-42f5-924d-0c2224a1f57e" (UID: "4298cde5-e51e-42f5-924d-0c2224a1f57e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:04.591439 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.591391 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-dshm" pod "0d9d315d-8875-4e45-8d34-2005ddd628d5" (UID: "0d9d315d-8875-4e45-8d34-2005ddd628d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:04.592538 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.592494 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d9d315d-8875-4e45-8d34-2005ddd628d5-tls-certs" pod "0d9d315d-8875-4e45-8d34-2005ddd628d5" (UID: "0d9d315d-8875-4e45-8d34-2005ddd628d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:04.592538 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.592502 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-dshm" pod "4298cde5-e51e-42f5-924d-0c2224a1f57e" (UID: "4298cde5-e51e-42f5-924d-0c2224a1f57e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:04.592990 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.592962 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d9d315d-8875-4e45-8d34-2005ddd628d5-kube-api-access-62mh5" pod "0d9d315d-8875-4e45-8d34-2005ddd628d5" (UID: "0d9d315d-8875-4e45-8d34-2005ddd628d5"). InnerVolumeSpecName "kube-api-access-62mh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:04.600829 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.600775 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-tls-certs" pod "193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" (UID: "193c8acd-7eb4-49f3-b2b3-0c398cdd9c02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:04.680139 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.680052 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4298cde5-e51e-42f5-924d-0c2224a1f57e-kube-api-access-qxtf2\" (UniqueName: \"kubernetes.io/projected/4298cde5-e51e-42f5-924d-0c2224a1f57e-kube-api-access-qxtf2\") pod \"4298cde5-e51e-42f5-924d-0c2224a1f57e\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " Oct 31 00:49:04.680139 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.680100 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-kube-api-access-nnfrr\" (UniqueName: \"kubernetes.io/projected/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-kube-api-access-nnfrr\") pod \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\" (UID: \"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02\") " Oct 31 00:49:04.680321 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.680139 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-model-cache\") pod \"0d9d315d-8875-4e45-8d34-2005ddd628d5\" (UID: \"0d9d315d-8875-4e45-8d34-2005ddd628d5\") " Oct 31 00:49:04.680321 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.680265 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-tmp-dir\") pod \"4298cde5-e51e-42f5-924d-0c2224a1f57e\" (UID: \"4298cde5-e51e-42f5-924d-0c2224a1f57e\") " Oct 31 00:49:04.680531 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.680481 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-model-cache" pod "0d9d315d-8875-4e45-8d34-2005ddd628d5" (UID: "0d9d315d-8875-4e45-8d34-2005ddd628d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:04.680592 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.680552 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-tmp-dir" pod "4298cde5-e51e-42f5-924d-0c2224a1f57e" (UID: "4298cde5-e51e-42f5-924d-0c2224a1f57e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:04.680677 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.680631 4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-dshm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.680742 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.680682 4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d9d315d-8875-4e45-8d34-2005ddd628d5-tls-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.680742 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.680693 4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-tls-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.680742 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.680703 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62mh5\" (UniqueName: \"kubernetes.io/projected/0d9d315d-8875-4e45-8d34-2005ddd628d5-kube-api-access-62mh5\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.680742 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.680712 4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-dshm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.680742 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.680721 4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4298cde5-e51e-42f5-924d-0c2224a1f57e-tls-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.680742 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.680729 4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-dshm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.682207 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.682182 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-kube-api-access-nnfrr" pod "193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" (UID: "193c8acd-7eb4-49f3-b2b3-0c398cdd9c02"). InnerVolumeSpecName "kube-api-access-nnfrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:04.682423 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.682404 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4298cde5-e51e-42f5-924d-0c2224a1f57e-kube-api-access-qxtf2" pod "4298cde5-e51e-42f5-924d-0c2224a1f57e" (UID: "4298cde5-e51e-42f5-924d-0c2224a1f57e"). InnerVolumeSpecName "kube-api-access-qxtf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:04.796673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.785590 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnfrr\" (UniqueName: \"kubernetes.io/projected/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02-kube-api-access-nnfrr\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.796673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.785651 4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d9d315d-8875-4e45-8d34-2005ddd628d5-model-cache\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.796673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.785669 4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4298cde5-e51e-42f5-924d-0c2224a1f57e-tmp-dir\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.796673 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.785685 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxtf2\" (UniqueName: \"kubernetes.io/projected/4298cde5-e51e-42f5-924d-0c2224a1f57e-kube-api-access-qxtf2\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.864693 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.861336 4519 generic.go:356] "Generic (PLEG): container finished" podID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerID="9c2b9907c8b36a44e201bf4ecc634088534a7d3564c31aad87df5fd19e7152d7" exitCode=0 Oct 31 00:49:04.864693 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.861492 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" event={"ID":"4298cde5-e51e-42f5-924d-0c2224a1f57e","Type":"ContainerDied","Data":"9c2b9907c8b36a44e201bf4ecc634088534a7d3564c31aad87df5fd19e7152d7"} Oct 31 00:49:04.864693 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.861528 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" event={"ID":"4298cde5-e51e-42f5-924d-0c2224a1f57e","Type":"ContainerDied","Data":"fe5219450cffa3a186a29335ee8ac1de16f0b9d0ec45cbd42e20fb1b2283145b"} Oct 31 00:49:04.864693 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.861549 4519 scope.go:122] "RemoveContainer" containerID="9c2b9907c8b36a44e201bf4ecc634088534a7d3564c31aad87df5fd19e7152d7" Oct 31 00:49:04.864693 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.861711 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz" Oct 31 00:49:04.869968 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.869943 4519 generic.go:356] "Generic (PLEG): container finished" podID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerID="73ef214360d8f16a916d2226135eb5455b1d62b7336fc626ee6ae391624265bb" exitCode=0 Oct 31 00:49:04.870113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.870040 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" event={"ID":"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02","Type":"ContainerDied","Data":"73ef214360d8f16a916d2226135eb5455b1d62b7336fc626ee6ae391624265bb"} Oct 31 00:49:04.870113 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.870075 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" event={"ID":"193c8acd-7eb4-49f3-b2b3-0c398cdd9c02","Type":"ContainerDied","Data":"f9502898d7bae49ef1fccc67011fcdb8014f625c919a509d548359b926988d4f"} Oct 31 00:49:04.870231 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.870136 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8" Oct 31 00:49:04.874886 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.874854 4519 generic.go:356] "Generic (PLEG): container finished" podID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerID="1db6086e867eafebe67d526fded02d3390ff41791870007fb666a2e7f5e84546" exitCode=0 Oct 31 00:49:04.875027 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.874910 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" event={"ID":"0d9d315d-8875-4e45-8d34-2005ddd628d5","Type":"ContainerDied","Data":"1db6086e867eafebe67d526fded02d3390ff41791870007fb666a2e7f5e84546"} Oct 31 00:49:04.875027 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.874943 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" event={"ID":"0d9d315d-8875-4e45-8d34-2005ddd628d5","Type":"ContainerDied","Data":"c87e74c44069ac17e535af8c709f30400d7b331538372938413227380d8e632b"} Oct 31 00:49:04.875027 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.875010 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl" Oct 31 00:49:04.877556 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.877520 4519 scope.go:122] "RemoveContainer" containerID="9c2b9907c8b36a44e201bf4ecc634088534a7d3564c31aad87df5fd19e7152d7" Oct 31 00:49:04.877929 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:04.877899 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c2b9907c8b36a44e201bf4ecc634088534a7d3564c31aad87df5fd19e7152d7\": container with ID starting with 9c2b9907c8b36a44e201bf4ecc634088534a7d3564c31aad87df5fd19e7152d7 not found: ID does not exist" containerID="9c2b9907c8b36a44e201bf4ecc634088534a7d3564c31aad87df5fd19e7152d7" Oct 31 00:49:04.878013 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.877940 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c2b9907c8b36a44e201bf4ecc634088534a7d3564c31aad87df5fd19e7152d7"} err="failed to get container status \"9c2b9907c8b36a44e201bf4ecc634088534a7d3564c31aad87df5fd19e7152d7\": rpc error: code = NotFound desc = could not find container \"9c2b9907c8b36a44e201bf4ecc634088534a7d3564c31aad87df5fd19e7152d7\": container with ID starting with 9c2b9907c8b36a44e201bf4ecc634088534a7d3564c31aad87df5fd19e7152d7 not found: ID does not exist" Oct 31 00:49:04.878013 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.877966 4519 scope.go:122] "RemoveContainer" containerID="73ef214360d8f16a916d2226135eb5455b1d62b7336fc626ee6ae391624265bb" Oct 31 00:49:04.941009 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.940929 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl"] Oct 31 00:49:04.950937 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.950905 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/simulator-w2-b433e6-kserve-bb7846f8d-6lmbl"] Oct 31 00:49:04.955803 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.955780 4519 scope.go:122] "RemoveContainer" containerID="73ef214360d8f16a916d2226135eb5455b1d62b7336fc626ee6ae391624265bb" Oct 31 00:49:04.956512 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:04.956351 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ef214360d8f16a916d2226135eb5455b1d62b7336fc626ee6ae391624265bb\": container with ID starting with 73ef214360d8f16a916d2226135eb5455b1d62b7336fc626ee6ae391624265bb not found: ID does not exist" containerID="73ef214360d8f16a916d2226135eb5455b1d62b7336fc626ee6ae391624265bb" Oct 31 00:49:04.956512 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.956389 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ef214360d8f16a916d2226135eb5455b1d62b7336fc626ee6ae391624265bb"} err="failed to get container status \"73ef214360d8f16a916d2226135eb5455b1d62b7336fc626ee6ae391624265bb\": rpc error: code = NotFound desc = could not find container \"73ef214360d8f16a916d2226135eb5455b1d62b7336fc626ee6ae391624265bb\": container with ID starting with 73ef214360d8f16a916d2226135eb5455b1d62b7336fc626ee6ae391624265bb not found: ID does not exist" Oct 31 00:49:04.956512 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.956415 4519 scope.go:122] "RemoveContainer" containerID="1db6086e867eafebe67d526fded02d3390ff41791870007fb666a2e7f5e84546" Oct 31 00:49:04.963050 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.963026 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8"] Oct 31 00:49:04.965887 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.965865 4519 scope.go:122] "RemoveContainer" containerID="1db6086e867eafebe67d526fded02d3390ff41791870007fb666a2e7f5e84546" Oct 31 00:49:04.966197 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:04.966179 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db6086e867eafebe67d526fded02d3390ff41791870007fb666a2e7f5e84546\": container with ID starting with 1db6086e867eafebe67d526fded02d3390ff41791870007fb666a2e7f5e84546 not found: ID does not exist" containerID="1db6086e867eafebe67d526fded02d3390ff41791870007fb666a2e7f5e84546" Oct 31 00:49:04.966266 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.966207 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db6086e867eafebe67d526fded02d3390ff41791870007fb666a2e7f5e84546"} err="failed to get container status \"1db6086e867eafebe67d526fded02d3390ff41791870007fb666a2e7f5e84546\": rpc error: code = NotFound desc = could not find container \"1db6086e867eafebe67d526fded02d3390ff41791870007fb666a2e7f5e84546\": container with ID starting with 1db6086e867eafebe67d526fded02d3390ff41791870007fb666a2e7f5e84546 not found: ID does not exist" Oct 31 00:49:04.971533 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:04.971506 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/distinct-w2-b433e6-kserve-7656d7bb48-xcfh8"] Oct 31 00:49:05.003440 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.003404 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz"] Oct 31 00:49:05.011523 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.011494 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b433e6/premium-w2-b433e6-kserve-7d7d48597c-lddlz"] Oct 31 00:49:05.067820 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.067790 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:05.100565 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.100539 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" Oct 31 00:49:05.293158 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.293124 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76bbc2e-d9b0-4ac4-a09d-91b609760c16-kube-api-access-m2gqk\" (UniqueName: \"kubernetes.io/projected/f76bbc2e-d9b0-4ac4-a09d-91b609760c16-kube-api-access-m2gqk\") pod \"f76bbc2e-d9b0-4ac4-a09d-91b609760c16\" (UID: \"f76bbc2e-d9b0-4ac4-a09d-91b609760c16\") " Oct 31 00:49:05.293316 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.293272 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f76bbc2e-d9b0-4ac4-a09d-91b609760c16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76bbc2e-d9b0-4ac4-a09d-91b609760c16-plugins-config-volume\") pod \"f76bbc2e-d9b0-4ac4-a09d-91b609760c16\" (UID: \"f76bbc2e-d9b0-4ac4-a09d-91b609760c16\") " Oct 31 00:49:05.293672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.293620 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76bbc2e-d9b0-4ac4-a09d-91b609760c16-plugins-config-volume" pod "f76bbc2e-d9b0-4ac4-a09d-91b609760c16" (UID: "f76bbc2e-d9b0-4ac4-a09d-91b609760c16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:05.295401 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.295383 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76bbc2e-d9b0-4ac4-a09d-91b609760c16-kube-api-access-m2gqk" pod "f76bbc2e-d9b0-4ac4-a09d-91b609760c16" (UID: "f76bbc2e-d9b0-4ac4-a09d-91b609760c16"). InnerVolumeSpecName "kube-api-access-m2gqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:05.394565 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.394528 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2gqk\" (UniqueName: \"kubernetes.io/projected/f76bbc2e-d9b0-4ac4-a09d-91b609760c16-kube-api-access-m2gqk\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.394565 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.394561 4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76bbc2e-d9b0-4ac4-a09d-91b609760c16-plugins-config-volume\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.707609 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.707577 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:49:05.794124 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.794101 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:49:05.797579 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.797550 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-envoy\") pod \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " Oct 31 00:49:05.797738 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.797628 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-credential-socket\") pod \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " Oct 31 00:49:05.797738 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.797685 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-workload-certs\") pod \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " Oct 31 00:49:05.797738 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.797744 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-workload-socket\") pod \"4edab197-6a3f-457a-acab-82d4cebc7b0f\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " Oct 31 00:49:05.798003 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.797791 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-podinfo\") pod \"4edab197-6a3f-457a-acab-82d4cebc7b0f\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " Oct 31 00:49:05.798003 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.797826 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-credential-socket\") pod \"4edab197-6a3f-457a-acab-82d4cebc7b0f\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " Oct 31 00:49:05.798003 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.797876 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdbf7b30-65d4-4462-af10-05cdd5015ce3-kube-api-access-dvbk4\" (UniqueName: \"kubernetes.io/projected/cdbf7b30-65d4-4462-af10-05cdd5015ce3-kube-api-access-dvbk4\") pod \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " Oct 31 00:49:05.798003 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.797912 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-token\" (UniqueName: \"kubernetes.io/projected/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-token\") pod \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " Oct 31 00:49:05.798003 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.797943 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-envoy\") pod \"4edab197-6a3f-457a-acab-82d4cebc7b0f\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " Oct 31 00:49:05.798258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798003 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-workload-certs\") pod \"4edab197-6a3f-457a-acab-82d4cebc7b0f\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " Oct 31 00:49:05.798258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798030 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-token\" (UniqueName: \"kubernetes.io/projected/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-token\") pod \"4edab197-6a3f-457a-acab-82d4cebc7b0f\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " Oct 31 00:49:05.798258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798076 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-podinfo\") pod \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " Oct 31 00:49:05.798258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798090 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-workload-certs" pod "cdbf7b30-65d4-4462-af10-05cdd5015ce3" (UID: "cdbf7b30-65d4-4462-af10-05cdd5015ce3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:05.798258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798104 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-data\") pod \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " Oct 31 00:49:05.798258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798140 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-workload-socket\") pod \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " Oct 31 00:49:05.798258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798186 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istiod-ca-cert\") pod \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\" (UID: \"cdbf7b30-65d4-4462-af10-05cdd5015ce3\") " Oct 31 00:49:05.798258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798245 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4edab197-6a3f-457a-acab-82d4cebc7b0f-kube-api-access-brgmx\" (UniqueName: \"kubernetes.io/projected/4edab197-6a3f-457a-acab-82d4cebc7b0f-kube-api-access-brgmx\") pod \"4edab197-6a3f-457a-acab-82d4cebc7b0f\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " Oct 31 00:49:05.798689 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798286 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-data\") pod \"4edab197-6a3f-457a-acab-82d4cebc7b0f\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " Oct 31 00:49:05.798689 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798315 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4edab197-6a3f-457a-acab-82d4cebc7b0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4edab197-6a3f-457a-acab-82d4cebc7b0f-istiod-ca-cert\") pod \"4edab197-6a3f-457a-acab-82d4cebc7b0f\" (UID: \"4edab197-6a3f-457a-acab-82d4cebc7b0f\") " Oct 31 00:49:05.798689 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798314 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-workload-socket" pod "4edab197-6a3f-457a-acab-82d4cebc7b0f" (UID: "4edab197-6a3f-457a-acab-82d4cebc7b0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:05.798689 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798375 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-workload-certs" pod "4edab197-6a3f-457a-acab-82d4cebc7b0f" (UID: "4edab197-6a3f-457a-acab-82d4cebc7b0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:05.798689 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798528 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-workload-socket" pod "cdbf7b30-65d4-4462-af10-05cdd5015ce3" (UID: "cdbf7b30-65d4-4462-af10-05cdd5015ce3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:05.798957 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798775 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.798957 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798795 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.798957 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798817 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.798957 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798832 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.798957 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798837 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4edab197-6a3f-457a-acab-82d4cebc7b0f-istiod-ca-cert" pod "4edab197-6a3f-457a-acab-82d4cebc7b0f" (UID: "4edab197-6a3f-457a-acab-82d4cebc7b0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:05.799236 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.799024 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-data" pod "cdbf7b30-65d4-4462-af10-05cdd5015ce3" (UID: "cdbf7b30-65d4-4462-af10-05cdd5015ce3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:05.799236 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.798523 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-credential-socket" pod "4edab197-6a3f-457a-acab-82d4cebc7b0f" (UID: "4edab197-6a3f-457a-acab-82d4cebc7b0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:05.799340 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.799263 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istiod-ca-cert" pod "cdbf7b30-65d4-4462-af10-05cdd5015ce3" (UID: "cdbf7b30-65d4-4462-af10-05cdd5015ce3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:05.799340 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.799279 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-credential-socket" pod "cdbf7b30-65d4-4462-af10-05cdd5015ce3" (UID: "cdbf7b30-65d4-4462-af10-05cdd5015ce3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:05.799480 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.799459 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-data" pod "4edab197-6a3f-457a-acab-82d4cebc7b0f" (UID: "4edab197-6a3f-457a-acab-82d4cebc7b0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:05.800240 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.800208 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-envoy" pod "cdbf7b30-65d4-4462-af10-05cdd5015ce3" (UID: "cdbf7b30-65d4-4462-af10-05cdd5015ce3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:05.801131 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.801103 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbf7b30-65d4-4462-af10-05cdd5015ce3-kube-api-access-dvbk4" pod "cdbf7b30-65d4-4462-af10-05cdd5015ce3" (UID: "cdbf7b30-65d4-4462-af10-05cdd5015ce3"). InnerVolumeSpecName "kube-api-access-dvbk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:05.801229 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.801188 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-podinfo" pod "cdbf7b30-65d4-4462-af10-05cdd5015ce3" (UID: "cdbf7b30-65d4-4462-af10-05cdd5015ce3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:05.801229 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.801214 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4edab197-6a3f-457a-acab-82d4cebc7b0f-kube-api-access-brgmx" pod "4edab197-6a3f-457a-acab-82d4cebc7b0f" (UID: "4edab197-6a3f-457a-acab-82d4cebc7b0f"). InnerVolumeSpecName "kube-api-access-brgmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:05.801352 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.801295 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-podinfo" pod "4edab197-6a3f-457a-acab-82d4cebc7b0f" (UID: "4edab197-6a3f-457a-acab-82d4cebc7b0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:05.801468 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.801448 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-token" pod "4edab197-6a3f-457a-acab-82d4cebc7b0f" (UID: "4edab197-6a3f-457a-acab-82d4cebc7b0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:05.801701 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.801687 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-token" pod "cdbf7b30-65d4-4462-af10-05cdd5015ce3" (UID: "cdbf7b30-65d4-4462-af10-05cdd5015ce3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:05.802335 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.802321 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-envoy" pod "4edab197-6a3f-457a-acab-82d4cebc7b0f" (UID: "4edab197-6a3f-457a-acab-82d4cebc7b0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:05.886177 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.886099 4519 generic.go:356] "Generic (PLEG): container finished" podID="f76bbc2e-d9b0-4ac4-a09d-91b609760c16" containerID="e8b4494e095cdf344c1f883571ca176a2ed607955140e47b6e1b40963ffb0d6a" exitCode=0 Oct 31 00:49:05.886177 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.886150 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" Oct 31 00:49:05.886394 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.886183 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" event={"ID":"f76bbc2e-d9b0-4ac4-a09d-91b609760c16","Type":"ContainerDied","Data":"e8b4494e095cdf344c1f883571ca176a2ed607955140e47b6e1b40963ffb0d6a"} Oct 31 00:49:05.886394 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.886235 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld" event={"ID":"f76bbc2e-d9b0-4ac4-a09d-91b609760c16","Type":"ContainerDied","Data":"3996541fd0b7239e105f18d414e70e3bfb760891a0485cb6cc0152286cd7ddee"} Oct 31 00:49:05.886394 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.886264 4519 scope.go:122] "RemoveContainer" containerID="e8b4494e095cdf344c1f883571ca176a2ed607955140e47b6e1b40963ffb0d6a" Oct 31 00:49:05.888172 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.888140 4519 generic.go:356] "Generic (PLEG): container finished" podID="cdbf7b30-65d4-4462-af10-05cdd5015ce3" containerID="727939f6d98ac00afaa2991e4b057de2077f56d8f187b71aabc26ac40a1d97fb" exitCode=0 Oct 31 00:49:05.888316 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.888196 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" Oct 31 00:49:05.888316 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.888223 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" event={"ID":"cdbf7b30-65d4-4462-af10-05cdd5015ce3","Type":"ContainerDied","Data":"727939f6d98ac00afaa2991e4b057de2077f56d8f187b71aabc26ac40a1d97fb"} Oct 31 00:49:05.888316 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.888264 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv" event={"ID":"cdbf7b30-65d4-4462-af10-05cdd5015ce3","Type":"ContainerDied","Data":"07dfc0550767c8439c5eae61cb9dd3f627e735b180aa1054ba4b607a49cb3805"} Oct 31 00:49:05.894687 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.892840 4519 generic.go:356] "Generic (PLEG): container finished" podID="4edab197-6a3f-457a-acab-82d4cebc7b0f" containerID="6348482c530ea8af1d77c1fa72d36d3c3a545c2f1d3c44960310b79809c95142" exitCode=0 Oct 31 00:49:05.894687 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.892888 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" Oct 31 00:49:05.894687 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.892942 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" event={"ID":"4edab197-6a3f-457a-acab-82d4cebc7b0f","Type":"ContainerDied","Data":"6348482c530ea8af1d77c1fa72d36d3c3a545c2f1d3c44960310b79809c95142"} Oct 31 00:49:05.894687 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.892982 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq" event={"ID":"4edab197-6a3f-457a-acab-82d4cebc7b0f","Type":"ContainerDied","Data":"8b26dfb6d98df8cd5ae310489885c2a52671560af6289f54d36f68027dcb3c18"} Oct 31 00:49:05.899457 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.898687 4519 scope.go:122] "RemoveContainer" containerID="e8b4494e095cdf344c1f883571ca176a2ed607955140e47b6e1b40963ffb0d6a" Oct 31 00:49:05.899457 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:05.899252 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b4494e095cdf344c1f883571ca176a2ed607955140e47b6e1b40963ffb0d6a\": container with ID starting with e8b4494e095cdf344c1f883571ca176a2ed607955140e47b6e1b40963ffb0d6a not found: ID does not exist" containerID="e8b4494e095cdf344c1f883571ca176a2ed607955140e47b6e1b40963ffb0d6a" Oct 31 00:49:05.899457 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.899286 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b4494e095cdf344c1f883571ca176a2ed607955140e47b6e1b40963ffb0d6a"} err="failed to get container status \"e8b4494e095cdf344c1f883571ca176a2ed607955140e47b6e1b40963ffb0d6a\": rpc error: code = NotFound desc = could not find container \"e8b4494e095cdf344c1f883571ca176a2ed607955140e47b6e1b40963ffb0d6a\": container with ID starting with e8b4494e095cdf344c1f883571ca176a2ed607955140e47b6e1b40963ffb0d6a not found: ID does not exist" Oct 31 00:49:05.899457 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.899310 4519 scope.go:122] "RemoveContainer" containerID="727939f6d98ac00afaa2991e4b057de2077f56d8f187b71aabc26ac40a1d97fb" Oct 31 00:49:05.902318 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.902298 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.902486 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.902465 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brgmx\" (UniqueName: \"kubernetes.io/projected/4edab197-6a3f-457a-acab-82d4cebc7b0f-kube-api-access-brgmx\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.902563 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.902488 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.902563 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.902502 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4edab197-6a3f-457a-acab-82d4cebc7b0f-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.902563 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.902524 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.902563 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.902539 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.902563 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.902552 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.902923 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.902566 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.902923 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.902580 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvbk4\" (UniqueName: \"kubernetes.io/projected/cdbf7b30-65d4-4462-af10-05cdd5015ce3-kube-api-access-dvbk4\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.902923 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.902595 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.902923 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.902608 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.902923 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.902627 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4edab197-6a3f-457a-acab-82d4cebc7b0f-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.902923 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.902656 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.902923 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.902669 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbf7b30-65d4-4462-af10-05cdd5015ce3-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.912352 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.912325 4519 scope.go:122] "RemoveContainer" containerID="727939f6d98ac00afaa2991e4b057de2077f56d8f187b71aabc26ac40a1d97fb" Oct 31 00:49:05.912774 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:05.912738 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727939f6d98ac00afaa2991e4b057de2077f56d8f187b71aabc26ac40a1d97fb\": container with ID starting with 727939f6d98ac00afaa2991e4b057de2077f56d8f187b71aabc26ac40a1d97fb not found: ID does not exist" containerID="727939f6d98ac00afaa2991e4b057de2077f56d8f187b71aabc26ac40a1d97fb" Oct 31 00:49:05.912894 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.912787 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727939f6d98ac00afaa2991e4b057de2077f56d8f187b71aabc26ac40a1d97fb"} err="failed to get container status \"727939f6d98ac00afaa2991e4b057de2077f56d8f187b71aabc26ac40a1d97fb\": rpc error: code = NotFound desc = could not find container \"727939f6d98ac00afaa2991e4b057de2077f56d8f187b71aabc26ac40a1d97fb\": container with ID starting with 727939f6d98ac00afaa2991e4b057de2077f56d8f187b71aabc26ac40a1d97fb not found: ID does not exist" Oct 31 00:49:05.912894 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.912810 4519 scope.go:122] "RemoveContainer" containerID="6348482c530ea8af1d77c1fa72d36d3c3a545c2f1d3c44960310b79809c95142" Oct 31 00:49:05.920969 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.920938 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld"] Oct 31 00:49:05.924795 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.924770 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f99ac0-5b7c4bb88c-zqnld"] Oct 31 00:49:05.927026 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.927008 4519 scope.go:122] "RemoveContainer" containerID="6348482c530ea8af1d77c1fa72d36d3c3a545c2f1d3c44960310b79809c95142" Oct 31 00:49:05.928147 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:05.928104 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6348482c530ea8af1d77c1fa72d36d3c3a545c2f1d3c44960310b79809c95142\": container with ID starting with 6348482c530ea8af1d77c1fa72d36d3c3a545c2f1d3c44960310b79809c95142 not found: ID does not exist" containerID="6348482c530ea8af1d77c1fa72d36d3c3a545c2f1d3c44960310b79809c95142" Oct 31 00:49:05.928265 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.928139 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6348482c530ea8af1d77c1fa72d36d3c3a545c2f1d3c44960310b79809c95142"} err="failed to get container status \"6348482c530ea8af1d77c1fa72d36d3c3a545c2f1d3c44960310b79809c95142\": rpc error: code = NotFound desc = could not find container \"6348482c530ea8af1d77c1fa72d36d3c3a545c2f1d3c44960310b79809c95142\": container with ID starting with 6348482c530ea8af1d77c1fa72d36d3c3a545c2f1d3c44960310b79809c95142 not found: ID does not exist" Oct 31 00:49:05.940268 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.939806 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv"] Oct 31 00:49:05.943297 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.943276 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-ch5qv"] Oct 31 00:49:05.965568 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.965538 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq"] Oct 31 00:49:05.971439 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:05.971409 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fttnq"] Oct 31 00:49:06.035899 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.035871 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:49:06.103419 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.103338 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" path="/var/lib/kubelet/pods/0d9d315d-8875-4e45-8d34-2005ddd628d5/volumes" Oct 31 00:49:06.103825 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.103804 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-istio-data\") pod \"e5fc25c3-483d-4644-b774-815f60af2025\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " Oct 31 00:49:06.103900 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.103861 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-istio-envoy\") pod \"e5fc25c3-483d-4644-b774-815f60af2025\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " Oct 31 00:49:06.103957 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.103913 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5fc25c3-483d-4644-b774-815f60af2025-istio-token\" (UniqueName: \"kubernetes.io/projected/e5fc25c3-483d-4644-b774-815f60af2025-istio-token\") pod \"e5fc25c3-483d-4644-b774-815f60af2025\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " Oct 31 00:49:06.104017 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.103976 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-workload-certs\") pod \"e5fc25c3-483d-4644-b774-815f60af2025\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " Oct 31 00:49:06.104017 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.103984 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" path="/var/lib/kubelet/pods/193c8acd-7eb4-49f3-b2b3-0c398cdd9c02/volumes" Oct 31 00:49:06.104124 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.104044 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-workload-socket\") pod \"e5fc25c3-483d-4644-b774-815f60af2025\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " Oct 31 00:49:06.104124 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.104071 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5fc25c3-483d-4644-b774-815f60af2025-kube-api-access-2jnhs\" (UniqueName: \"kubernetes.io/projected/e5fc25c3-483d-4644-b774-815f60af2025-kube-api-access-2jnhs\") pod \"e5fc25c3-483d-4644-b774-815f60af2025\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " Oct 31 00:49:06.104124 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.104100 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-credential-socket\") pod \"e5fc25c3-483d-4644-b774-815f60af2025\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " Oct 31 00:49:06.104268 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.104121 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-istio-data" pod "e5fc25c3-483d-4644-b774-815f60af2025" (UID: "e5fc25c3-483d-4644-b774-815f60af2025"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:06.104268 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.104170 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5fc25c3-483d-4644-b774-815f60af2025-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5fc25c3-483d-4644-b774-815f60af2025-istiod-ca-cert\") pod \"e5fc25c3-483d-4644-b774-815f60af2025\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " Oct 31 00:49:06.104268 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.104240 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5fc25c3-483d-4644-b774-815f60af2025-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5fc25c3-483d-4644-b774-815f60af2025-istio-podinfo\") pod \"e5fc25c3-483d-4644-b774-815f60af2025\" (UID: \"e5fc25c3-483d-4644-b774-815f60af2025\") " Oct 31 00:49:06.104468 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.104342 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-workload-certs" pod "e5fc25c3-483d-4644-b774-815f60af2025" (UID: "e5fc25c3-483d-4644-b774-815f60af2025"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:06.104952 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.104861 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-credential-socket" pod "e5fc25c3-483d-4644-b774-815f60af2025" (UID: "e5fc25c3-483d-4644-b774-815f60af2025"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:06.104952 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.104876 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-workload-socket" pod "e5fc25c3-483d-4644-b774-815f60af2025" (UID: "e5fc25c3-483d-4644-b774-815f60af2025"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:06.104952 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.104901 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.104952 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.104924 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.104952 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.104941 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" path="/var/lib/kubelet/pods/4298cde5-e51e-42f5-924d-0c2224a1f57e/volumes" Oct 31 00:49:06.105309 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.104984 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5fc25c3-483d-4644-b774-815f60af2025-istiod-ca-cert" pod "e5fc25c3-483d-4644-b774-815f60af2025" (UID: "e5fc25c3-483d-4644-b774-815f60af2025"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:06.105543 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.105521 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4edab197-6a3f-457a-acab-82d4cebc7b0f" path="/var/lib/kubelet/pods/4edab197-6a3f-457a-acab-82d4cebc7b0f/volumes" Oct 31 00:49:06.106139 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.106117 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdbf7b30-65d4-4462-af10-05cdd5015ce3" path="/var/lib/kubelet/pods/cdbf7b30-65d4-4462-af10-05cdd5015ce3/volumes" Oct 31 00:49:06.107034 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.107011 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fc25c3-483d-4644-b774-815f60af2025-kube-api-access-2jnhs" pod "e5fc25c3-483d-4644-b774-815f60af2025" (UID: "e5fc25c3-483d-4644-b774-815f60af2025"). InnerVolumeSpecName "kube-api-access-2jnhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:06.107133 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.107052 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5fc25c3-483d-4644-b774-815f60af2025-istio-podinfo" pod "e5fc25c3-483d-4644-b774-815f60af2025" (UID: "e5fc25c3-483d-4644-b774-815f60af2025"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:06.107133 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.107089 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f76bbc2e-d9b0-4ac4-a09d-91b609760c16" path="/var/lib/kubelet/pods/f76bbc2e-d9b0-4ac4-a09d-91b609760c16/volumes" Oct 31 00:49:06.107251 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.107166 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fc25c3-483d-4644-b774-815f60af2025-istio-token" pod "e5fc25c3-483d-4644-b774-815f60af2025" (UID: "e5fc25c3-483d-4644-b774-815f60af2025"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:06.107476 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.107446 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-istio-envoy" pod "e5fc25c3-483d-4644-b774-815f60af2025" (UID: "e5fc25c3-483d-4644-b774-815f60af2025"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:06.206421 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.206384 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.206421 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.206418 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5fc25c3-483d-4644-b774-815f60af2025-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.206661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.206432 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.206661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.206442 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jnhs\" (UniqueName: \"kubernetes.io/projected/e5fc25c3-483d-4644-b774-815f60af2025-kube-api-access-2jnhs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.206661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.206451 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5fc25c3-483d-4644-b774-815f60af2025-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.206661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.206461 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5fc25c3-483d-4644-b774-815f60af2025-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.206661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.206469 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5fc25c3-483d-4644-b774-815f60af2025-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.907630 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.907595 4519 generic.go:356] "Generic (PLEG): container finished" podID="e5fc25c3-483d-4644-b774-815f60af2025" containerID="6fa1b78056d929bb107371a158e3918f3ef5908eb02a375be0e1f3a5935d4daa" exitCode=0 Oct 31 00:49:06.908131 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.907676 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" Oct 31 00:49:06.908131 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.907688 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" event={"ID":"e5fc25c3-483d-4644-b774-815f60af2025","Type":"ContainerDied","Data":"6fa1b78056d929bb107371a158e3918f3ef5908eb02a375be0e1f3a5935d4daa"} Oct 31 00:49:06.908131 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.907751 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w" event={"ID":"e5fc25c3-483d-4644-b774-815f60af2025","Type":"ContainerDied","Data":"fe405f4ce8d2fca74caa42ea587ffb71ad8b9ae791dd3c62c1675d5512a72883"} Oct 31 00:49:06.908131 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.907782 4519 scope.go:122] "RemoveContainer" containerID="6fa1b78056d929bb107371a158e3918f3ef5908eb02a375be0e1f3a5935d4daa" Oct 31 00:49:06.917537 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.917519 4519 scope.go:122] "RemoveContainer" containerID="6fa1b78056d929bb107371a158e3918f3ef5908eb02a375be0e1f3a5935d4daa" Oct 31 00:49:06.917850 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:06.917831 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa1b78056d929bb107371a158e3918f3ef5908eb02a375be0e1f3a5935d4daa\": container with ID starting with 6fa1b78056d929bb107371a158e3918f3ef5908eb02a375be0e1f3a5935d4daa not found: ID does not exist" containerID="6fa1b78056d929bb107371a158e3918f3ef5908eb02a375be0e1f3a5935d4daa" Oct 31 00:49:06.917911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.917857 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa1b78056d929bb107371a158e3918f3ef5908eb02a375be0e1f3a5935d4daa"} err="failed to get container status \"6fa1b78056d929bb107371a158e3918f3ef5908eb02a375be0e1f3a5935d4daa\": rpc error: code = NotFound desc = could not find container \"6fa1b78056d929bb107371a158e3918f3ef5908eb02a375be0e1f3a5935d4daa\": container with ID starting with 6fa1b78056d929bb107371a158e3918f3ef5908eb02a375be0e1f3a5935d4daa not found: ID does not exist" Oct 31 00:49:06.935356 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.935324 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w"] Oct 31 00:49:06.944277 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:06.944244 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f99ac0-openshift-default-84bc7fcc97-fcq7w"] Oct 31 00:49:08.101550 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:08.101514 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5fc25c3-483d-4644-b774-815f60af2025" path="/var/lib/kubelet/pods/e5fc25c3-483d-4644-b774-815f60af2025/volumes" Oct 31 00:49:10.568859 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:10.568824 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6"] Oct 31 00:49:10.569483 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:10.569089 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" podUID="9c1e90c3-947d-41f2-a4bb-fca7f4b54c40" containerName="maas-api" containerID="cri-o://b4ab2e84369bc88094854fa22ed3bfd8d12102fde92fa14544d43aadc58d21fd" gracePeriod=30 Oct 31 00:49:10.610174 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:10.610135 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:10.819981 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:10.816115 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:10.918576 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:10.918298 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:10.939192 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:10.938945 4519 generic.go:356] "Generic (PLEG): container finished" podID="9c1e90c3-947d-41f2-a4bb-fca7f4b54c40" containerID="b4ab2e84369bc88094854fa22ed3bfd8d12102fde92fa14544d43aadc58d21fd" exitCode=0 Oct 31 00:49:10.939192 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:10.939113 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" event={"ID":"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40","Type":"ContainerDied","Data":"b4ab2e84369bc88094854fa22ed3bfd8d12102fde92fa14544d43aadc58d21fd"} Oct 31 00:49:10.986006 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:10.985983 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:49:11.073627 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.073528 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:11.076378 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.075357 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-maas-api-tls\") pod \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\" (UID: \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\") " Oct 31 00:49:11.076378 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.075411 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-metrics-certs\") pod \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\" (UID: \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\") " Oct 31 00:49:11.076378 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.075486 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-openshift-service-ca\") pod \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\" (UID: \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\") " Oct 31 00:49:11.076378 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.075675 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-kube-api-access-rggkt\" (UniqueName: \"kubernetes.io/projected/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-kube-api-access-rggkt\") pod \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\" (UID: \"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40\") " Oct 31 00:49:11.077193 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.077146 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-openshift-service-ca" pod "9c1e90c3-947d-41f2-a4bb-fca7f4b54c40" (UID: "9c1e90c3-947d-41f2-a4bb-fca7f4b54c40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:11.079169 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.079126 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-kube-api-access-rggkt" pod "9c1e90c3-947d-41f2-a4bb-fca7f4b54c40" (UID: "9c1e90c3-947d-41f2-a4bb-fca7f4b54c40"). InnerVolumeSpecName "kube-api-access-rggkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:11.082564 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.082488 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-maas-api-tls" pod "9c1e90c3-947d-41f2-a4bb-fca7f4b54c40" (UID: "9c1e90c3-947d-41f2-a4bb-fca7f4b54c40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:11.084236 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.084204 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-metrics-certs" pod "9c1e90c3-947d-41f2-a4bb-fca7f4b54c40" (UID: "9c1e90c3-947d-41f2-a4bb-fca7f4b54c40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:11.192682 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.183500 4519 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-openshift-service-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:11.192682 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.183542 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rggkt\" (UniqueName: \"kubernetes.io/projected/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-kube-api-access-rggkt\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:11.192682 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.183557 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-maas-api-tls\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:11.192682 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.183571 4519 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40-metrics-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:11.194002 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.193821 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:11.366219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.366138 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:11.466189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.466160 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:11.638823 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.633739 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:11.875629 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.875590 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9"] Oct 31 00:49:11.875917 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.875886 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" podUID="fddc64cf-c911-4921-b176-a7c119cdd118" containerName="istio-proxy" containerID="cri-o://9fed18f5c0ad267bdf0f1cd44b9a7f985195622529b61a5fefff5b6c56dccff0" gracePeriod=30 Oct 31 00:49:11.879582 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.879554 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn"] Oct 31 00:49:11.879844 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.879818 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" containerID="cri-o://a0e48461a68564f3f5616b4e84451907a7dca742aaa5a86a9e9b91ad867ad0d8" gracePeriod=30 Oct 31 00:49:11.886047 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.886023 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:11.894286 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.894211 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k"] Oct 31 00:49:11.894470 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.894445 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" podUID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" containerName="istio-proxy" containerID="cri-o://0a7dec6811c1d96b62f3a9d7b834bef41758df021a32c6f1e4dc9059099272f9" gracePeriod=30 Oct 31 00:49:11.946530 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.946490 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" event={"ID":"9c1e90c3-947d-41f2-a4bb-fca7f4b54c40","Type":"ContainerDied","Data":"f8f88f723be8c1e3a0f91537ee71413c3c8f512ae106f82b045c4494e4bb0a00"} Oct 31 00:49:11.946530 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.946526 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6" Oct 31 00:49:11.946823 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:11.946556 4519 scope.go:122] "RemoveContainer" containerID="b4ab2e84369bc88094854fa22ed3bfd8d12102fde92fa14544d43aadc58d21fd" Oct 31 00:49:12.066374 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:12.066323 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6"] Oct 31 00:49:12.075265 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:12.075230 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b433e6-5596899d8-j6xj6"] Oct 31 00:49:12.101841 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:12.101804 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1e90c3-947d-41f2-a4bb-fca7f4b54c40" path="/var/lib/kubelet/pods/9c1e90c3-947d-41f2-a4bb-fca7f4b54c40/volumes" Oct 31 00:49:13.238693 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.238666 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" Oct 31 00:49:13.363702 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.363590 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b03043a-75b4-404a-800f-d71f33a04108-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b03043a-75b4-404a-800f-d71f33a04108-plugins-config-volume\") pod \"6b03043a-75b4-404a-800f-d71f33a04108\" (UID: \"6b03043a-75b4-404a-800f-d71f33a04108\") " Oct 31 00:49:13.363854 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.363819 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b03043a-75b4-404a-800f-d71f33a04108-kube-api-access-ptszr\" (UniqueName: \"kubernetes.io/projected/6b03043a-75b4-404a-800f-d71f33a04108-kube-api-access-ptszr\") pod \"6b03043a-75b4-404a-800f-d71f33a04108\" (UID: \"6b03043a-75b4-404a-800f-d71f33a04108\") " Oct 31 00:49:13.364078 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.364056 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b03043a-75b4-404a-800f-d71f33a04108-plugins-config-volume" pod "6b03043a-75b4-404a-800f-d71f33a04108" (UID: "6b03043a-75b4-404a-800f-d71f33a04108"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:13.364187 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.364174 4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b03043a-75b4-404a-800f-d71f33a04108-plugins-config-volume\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:13.365908 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.365879 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b03043a-75b4-404a-800f-d71f33a04108-kube-api-access-ptszr" pod "6b03043a-75b4-404a-800f-d71f33a04108" (UID: "6b03043a-75b4-404a-800f-d71f33a04108"). InnerVolumeSpecName "kube-api-access-ptszr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:13.464966 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.464935 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptszr\" (UniqueName: \"kubernetes.io/projected/6b03043a-75b4-404a-800f-d71f33a04108-kube-api-access-ptszr\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:13.549015 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.548980 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:13.772323 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.772289 4519 patch_prober.go:29] interesting pod/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:49:13.772485 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.772360 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:49:13.960770 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.960730 4519 generic.go:356] "Generic (PLEG): container finished" podID="6b03043a-75b4-404a-800f-d71f33a04108" containerID="7544849631b0f15a08d6e0b17f5c84594a76b78e0111356aebde942dad29b6f4" exitCode=0 Oct 31 00:49:13.960970 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.960788 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" Oct 31 00:49:13.960970 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.960813 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" event={"ID":"6b03043a-75b4-404a-800f-d71f33a04108","Type":"ContainerDied","Data":"7544849631b0f15a08d6e0b17f5c84594a76b78e0111356aebde942dad29b6f4"} Oct 31 00:49:13.960970 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.960895 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k" event={"ID":"6b03043a-75b4-404a-800f-d71f33a04108","Type":"ContainerDied","Data":"8869353d8c0f67f8e789008ead84f4c24c6e650fcc9c4fc1fef6d63d59034896"} Oct 31 00:49:13.960970 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.960918 4519 scope.go:122] "RemoveContainer" containerID="7544849631b0f15a08d6e0b17f5c84594a76b78e0111356aebde942dad29b6f4" Oct 31 00:49:13.971538 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.971515 4519 scope.go:122] "RemoveContainer" containerID="7544849631b0f15a08d6e0b17f5c84594a76b78e0111356aebde942dad29b6f4" Oct 31 00:49:13.971898 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:13.971878 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7544849631b0f15a08d6e0b17f5c84594a76b78e0111356aebde942dad29b6f4\": container with ID starting with 7544849631b0f15a08d6e0b17f5c84594a76b78e0111356aebde942dad29b6f4 not found: ID does not exist" containerID="7544849631b0f15a08d6e0b17f5c84594a76b78e0111356aebde942dad29b6f4" Oct 31 00:49:13.971959 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.971909 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7544849631b0f15a08d6e0b17f5c84594a76b78e0111356aebde942dad29b6f4"} err="failed to get container status \"7544849631b0f15a08d6e0b17f5c84594a76b78e0111356aebde942dad29b6f4\": rpc error: code = NotFound desc = could not find container \"7544849631b0f15a08d6e0b17f5c84594a76b78e0111356aebde942dad29b6f4\": container with ID starting with 7544849631b0f15a08d6e0b17f5c84594a76b78e0111356aebde942dad29b6f4 not found: ID does not exist" Oct 31 00:49:13.983747 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.983718 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k"] Oct 31 00:49:13.988360 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:13.988334 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5c2a03-74c7c69d47-fqf4k"] Oct 31 00:49:14.101827 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:14.101735 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b03043a-75b4-404a-800f-d71f33a04108" path="/var/lib/kubelet/pods/6b03043a-75b4-404a-800f-d71f33a04108/volumes" Oct 31 00:49:17.159288 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.159216 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:49:17.178616 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.178596 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:49:17.194152 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.194125 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:49:17.302274 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302238 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fddc64cf-c911-4921-b176-a7c119cdd118-istio-token\" (UniqueName: \"kubernetes.io/projected/fddc64cf-c911-4921-b176-a7c119cdd118-istio-token\") pod \"fddc64cf-c911-4921-b176-a7c119cdd118\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " Oct 31 00:49:17.302428 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302300 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-workload-certs\") pod \"b22ec283-637b-4dd0-add9-76777f64e77c\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " Oct 31 00:49:17.302428 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302338 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istiod-ca-cert\") pod \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " Oct 31 00:49:17.302428 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302375 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fddc64cf-c911-4921-b176-a7c119cdd118-kube-api-access-czbx5\" (UniqueName: \"kubernetes.io/projected/fddc64cf-c911-4921-b176-a7c119cdd118-kube-api-access-czbx5\") pod \"fddc64cf-c911-4921-b176-a7c119cdd118\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " Oct 31 00:49:17.302428 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302394 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-workload-socket\") pod \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " Oct 31 00:49:17.302428 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302418 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b22ec283-637b-4dd0-add9-76777f64e77c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22ec283-637b-4dd0-add9-76777f64e77c-istiod-ca-cert\") pod \"b22ec283-637b-4dd0-add9-76777f64e77c\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " Oct 31 00:49:17.302737 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302438 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a11f261-4cc7-4ca9-8b5f-8920883108b4-kube-api-access-jvw9v\" (UniqueName: \"kubernetes.io/projected/6a11f261-4cc7-4ca9-8b5f-8920883108b4-kube-api-access-jvw9v\") pod \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " Oct 31 00:49:17.302737 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302475 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-credential-socket\") pod \"b22ec283-637b-4dd0-add9-76777f64e77c\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " Oct 31 00:49:17.302737 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302504 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-workload-socket\") pod \"fddc64cf-c911-4921-b176-a7c119cdd118\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " Oct 31 00:49:17.302737 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302538 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-istio-envoy\") pod \"b22ec283-637b-4dd0-add9-76777f64e77c\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " Oct 31 00:49:17.302737 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302567 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-workload-certs" pod "b22ec283-637b-4dd0-add9-76777f64e77c" (UID: "b22ec283-637b-4dd0-add9-76777f64e77c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:17.302737 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302585 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-workload-certs\") pod \"fddc64cf-c911-4921-b176-a7c119cdd118\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " Oct 31 00:49:17.302737 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302680 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-credential-socket\") pod \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " Oct 31 00:49:17.302737 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302714 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fddc64cf-c911-4921-b176-a7c119cdd118-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fddc64cf-c911-4921-b176-a7c119cdd118-istiod-ca-cert\") pod \"fddc64cf-c911-4921-b176-a7c119cdd118\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " Oct 31 00:49:17.303226 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302743 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-token\" (UniqueName: \"kubernetes.io/projected/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-token\") pod \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " Oct 31 00:49:17.303226 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302773 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-credential-socket\") pod \"fddc64cf-c911-4921-b176-a7c119cdd118\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " Oct 31 00:49:17.303226 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302802 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-istio-data\") pod \"fddc64cf-c911-4921-b176-a7c119cdd118\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " Oct 31 00:49:17.303226 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302847 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-istio-envoy\") pod \"fddc64cf-c911-4921-b176-a7c119cdd118\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " Oct 31 00:49:17.303226 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302872 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22ec283-637b-4dd0-add9-76777f64e77c-kube-api-access-km4bt\" (UniqueName: \"kubernetes.io/projected/b22ec283-637b-4dd0-add9-76777f64e77c-kube-api-access-km4bt\") pod \"b22ec283-637b-4dd0-add9-76777f64e77c\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " Oct 31 00:49:17.303226 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302890 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b22ec283-637b-4dd0-add9-76777f64e77c-istiod-ca-cert" pod "b22ec283-637b-4dd0-add9-76777f64e77c" (UID: "b22ec283-637b-4dd0-add9-76777f64e77c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:17.303226 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302904 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-podinfo\") pod \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " Oct 31 00:49:17.303226 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302944 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-istio-data\") pod \"b22ec283-637b-4dd0-add9-76777f64e77c\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " Oct 31 00:49:17.303226 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.303041 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-data\") pod \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " Oct 31 00:49:17.303226 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.303071 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fddc64cf-c911-4921-b176-a7c119cdd118-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fddc64cf-c911-4921-b176-a7c119cdd118-istio-podinfo\") pod \"fddc64cf-c911-4921-b176-a7c119cdd118\" (UID: \"fddc64cf-c911-4921-b176-a7c119cdd118\") " Oct 31 00:49:17.303226 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.303100 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-workload-socket\") pod \"b22ec283-637b-4dd0-add9-76777f64e77c\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " Oct 31 00:49:17.303226 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.303131 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-workload-certs\") pod \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " Oct 31 00:49:17.303226 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.303164 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-envoy\") pod \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\" (UID: \"6a11f261-4cc7-4ca9-8b5f-8920883108b4\") " Oct 31 00:49:17.303226 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.303200 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b22ec283-637b-4dd0-add9-76777f64e77c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22ec283-637b-4dd0-add9-76777f64e77c-istio-podinfo\") pod \"b22ec283-637b-4dd0-add9-76777f64e77c\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " Oct 31 00:49:17.303226 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.303229 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22ec283-637b-4dd0-add9-76777f64e77c-istio-token\" (UniqueName: \"kubernetes.io/projected/b22ec283-637b-4dd0-add9-76777f64e77c-istio-token\") pod \"b22ec283-637b-4dd0-add9-76777f64e77c\" (UID: \"b22ec283-637b-4dd0-add9-76777f64e77c\") " Oct 31 00:49:17.303987 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.303783 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.303987 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.303805 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22ec283-637b-4dd0-add9-76777f64e77c-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.303987 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302872 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-workload-certs" pod "fddc64cf-c911-4921-b176-a7c119cdd118" (UID: "fddc64cf-c911-4921-b176-a7c119cdd118"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:17.303987 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.302889 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istiod-ca-cert" pod "6a11f261-4cc7-4ca9-8b5f-8920883108b4" (UID: "6a11f261-4cc7-4ca9-8b5f-8920883108b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:17.303987 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.303134 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-workload-socket" pod "6a11f261-4cc7-4ca9-8b5f-8920883108b4" (UID: "6a11f261-4cc7-4ca9-8b5f-8920883108b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:17.303987 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.303196 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-workload-socket" pod "fddc64cf-c911-4921-b176-a7c119cdd118" (UID: "fddc64cf-c911-4921-b176-a7c119cdd118"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:17.303987 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.303393 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-credential-socket" pod "fddc64cf-c911-4921-b176-a7c119cdd118" (UID: "fddc64cf-c911-4921-b176-a7c119cdd118"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:17.303987 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.303573 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-credential-socket" pod "6a11f261-4cc7-4ca9-8b5f-8920883108b4" (UID: "6a11f261-4cc7-4ca9-8b5f-8920883108b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:17.303987 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.303761 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-istio-data" pod "fddc64cf-c911-4921-b176-a7c119cdd118" (UID: "fddc64cf-c911-4921-b176-a7c119cdd118"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:17.303987 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.303825 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-credential-socket" pod "b22ec283-637b-4dd0-add9-76777f64e77c" (UID: "b22ec283-637b-4dd0-add9-76777f64e77c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:17.303987 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.303888 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fddc64cf-c911-4921-b176-a7c119cdd118-istiod-ca-cert" pod "fddc64cf-c911-4921-b176-a7c119cdd118" (UID: "fddc64cf-c911-4921-b176-a7c119cdd118"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:17.304530 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.304120 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-workload-certs" pod "6a11f261-4cc7-4ca9-8b5f-8920883108b4" (UID: "6a11f261-4cc7-4ca9-8b5f-8920883108b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:17.304530 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.304256 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-istio-data" pod "b22ec283-637b-4dd0-add9-76777f64e77c" (UID: "b22ec283-637b-4dd0-add9-76777f64e77c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:17.304530 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.304374 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-workload-socket" pod "b22ec283-637b-4dd0-add9-76777f64e77c" (UID: "b22ec283-637b-4dd0-add9-76777f64e77c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:17.304719 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.304613 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-data" pod "6a11f261-4cc7-4ca9-8b5f-8920883108b4" (UID: "6a11f261-4cc7-4ca9-8b5f-8920883108b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:17.305540 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.305511 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fddc64cf-c911-4921-b176-a7c119cdd118-istio-token" pod "fddc64cf-c911-4921-b176-a7c119cdd118" (UID: "fddc64cf-c911-4921-b176-a7c119cdd118"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:17.305701 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.305599 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fddc64cf-c911-4921-b176-a7c119cdd118-kube-api-access-czbx5" pod "fddc64cf-c911-4921-b176-a7c119cdd118" (UID: "fddc64cf-c911-4921-b176-a7c119cdd118"). InnerVolumeSpecName "kube-api-access-czbx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:17.306040 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.305977 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-istio-envoy" pod "fddc64cf-c911-4921-b176-a7c119cdd118" (UID: "fddc64cf-c911-4921-b176-a7c119cdd118"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:17.306728 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.306700 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22ec283-637b-4dd0-add9-76777f64e77c-istio-token" pod "b22ec283-637b-4dd0-add9-76777f64e77c" (UID: "b22ec283-637b-4dd0-add9-76777f64e77c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:17.306824 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.306755 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-envoy" pod "6a11f261-4cc7-4ca9-8b5f-8920883108b4" (UID: "6a11f261-4cc7-4ca9-8b5f-8920883108b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:17.306933 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.306894 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-istio-envoy" pod "b22ec283-637b-4dd0-add9-76777f64e77c" (UID: "b22ec283-637b-4dd0-add9-76777f64e77c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:17.306933 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.306923 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a11f261-4cc7-4ca9-8b5f-8920883108b4-kube-api-access-jvw9v" pod "6a11f261-4cc7-4ca9-8b5f-8920883108b4" (UID: "6a11f261-4cc7-4ca9-8b5f-8920883108b4"). InnerVolumeSpecName "kube-api-access-jvw9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:17.307094 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.307043 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-podinfo" pod "6a11f261-4cc7-4ca9-8b5f-8920883108b4" (UID: "6a11f261-4cc7-4ca9-8b5f-8920883108b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:17.307188 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.307165 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fddc64cf-c911-4921-b176-a7c119cdd118-istio-podinfo" pod "fddc64cf-c911-4921-b176-a7c119cdd118" (UID: "fddc64cf-c911-4921-b176-a7c119cdd118"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:17.307619 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.307597 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22ec283-637b-4dd0-add9-76777f64e77c-kube-api-access-km4bt" pod "b22ec283-637b-4dd0-add9-76777f64e77c" (UID: "b22ec283-637b-4dd0-add9-76777f64e77c"). InnerVolumeSpecName "kube-api-access-km4bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:17.307716 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.307619 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b22ec283-637b-4dd0-add9-76777f64e77c-istio-podinfo" pod "b22ec283-637b-4dd0-add9-76777f64e77c" (UID: "b22ec283-637b-4dd0-add9-76777f64e77c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:17.308060 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.308041 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-token" pod "6a11f261-4cc7-4ca9-8b5f-8920883108b4" (UID: "6a11f261-4cc7-4ca9-8b5f-8920883108b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:17.404883 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.404847 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.404883 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.404878 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.404883 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.404887 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.404897 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km4bt\" (UniqueName: \"kubernetes.io/projected/b22ec283-637b-4dd0-add9-76777f64e77c-kube-api-access-km4bt\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.404907 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.404915 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.404923 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.404932 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fddc64cf-c911-4921-b176-a7c119cdd118-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.404940 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.404947 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.404957 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.404964 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22ec283-637b-4dd0-add9-76777f64e77c-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.404972 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22ec283-637b-4dd0-add9-76777f64e77c-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.404979 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fddc64cf-c911-4921-b176-a7c119cdd118-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.404987 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.404996 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czbx5\" (UniqueName: \"kubernetes.io/projected/fddc64cf-c911-4921-b176-a7c119cdd118-kube-api-access-czbx5\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.405004 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.405012 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvw9v\" (UniqueName: \"kubernetes.io/projected/6a11f261-4cc7-4ca9-8b5f-8920883108b4-kube-api-access-jvw9v\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.405020 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.405029 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.405036 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22ec283-637b-4dd0-add9-76777f64e77c-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.405044 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fddc64cf-c911-4921-b176-a7c119cdd118-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.405054 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a11f261-4cc7-4ca9-8b5f-8920883108b4-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.405062 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fddc64cf-c911-4921-b176-a7c119cdd118-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.405128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.405070 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a11f261-4cc7-4ca9-8b5f-8920883108b4-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.540508 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:17.540463 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:49:17.613090 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.613060 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:17.712700 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.712657 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:17.970400 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.970363 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d"] Oct 31 00:49:17.970685 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.970656 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" containerID="cri-o://9e33b8afb51f75baebf9b8dc648d0c80d472fd199f646373c4dfff580adb7b54" gracePeriod=30 Oct 31 00:49:17.979629 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.979603 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm"] Oct 31 00:49:17.979871 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.979845 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" containerID="cri-o://fce4eb71bd9706f20425a86690448aa91797027e9680f4f323fb185d698fc2e4" gracePeriod=30 Oct 31 00:49:17.990169 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.990136 4519 generic.go:356] "Generic (PLEG): container finished" podID="b22ec283-637b-4dd0-add9-76777f64e77c" containerID="a0e48461a68564f3f5616b4e84451907a7dca742aaa5a86a9e9b91ad867ad0d8" exitCode=0 Oct 31 00:49:17.990314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.990175 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" event={"ID":"b22ec283-637b-4dd0-add9-76777f64e77c","Type":"ContainerDied","Data":"a0e48461a68564f3f5616b4e84451907a7dca742aaa5a86a9e9b91ad867ad0d8"} Oct 31 00:49:17.990314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.990201 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" Oct 31 00:49:17.990314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.990231 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn" event={"ID":"b22ec283-637b-4dd0-add9-76777f64e77c","Type":"ContainerDied","Data":"c8ca2caad342673a22cbda2f2150f38d47c13fe2f5153320f5d158ccbeca2260"} Oct 31 00:49:17.990314 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.990260 4519 scope.go:122] "RemoveContainer" containerID="a0e48461a68564f3f5616b4e84451907a7dca742aaa5a86a9e9b91ad867ad0d8" Oct 31 00:49:17.992975 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.992943 4519 generic.go:356] "Generic (PLEG): container finished" podID="fddc64cf-c911-4921-b176-a7c119cdd118" containerID="9fed18f5c0ad267bdf0f1cd44b9a7f985195622529b61a5fefff5b6c56dccff0" exitCode=0 Oct 31 00:49:17.993179 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.993072 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" event={"ID":"fddc64cf-c911-4921-b176-a7c119cdd118","Type":"ContainerDied","Data":"9fed18f5c0ad267bdf0f1cd44b9a7f985195622529b61a5fefff5b6c56dccff0"} Oct 31 00:49:17.993179 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.993137 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" event={"ID":"fddc64cf-c911-4921-b176-a7c119cdd118","Type":"ContainerDied","Data":"afac849f835049779142947fbc32c79a3654b999968e81217cd31b54a621cf19"} Oct 31 00:49:17.993409 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.993392 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9" Oct 31 00:49:17.995088 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.995065 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:17.995181 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.995124 4519 generic.go:356] "Generic (PLEG): container finished" podID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" containerID="0a7dec6811c1d96b62f3a9d7b834bef41758df021a32c6f1e4dc9059099272f9" exitCode=0 Oct 31 00:49:17.995181 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.995168 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" Oct 31 00:49:17.995290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.995214 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" event={"ID":"6a11f261-4cc7-4ca9-8b5f-8920883108b4","Type":"ContainerDied","Data":"0a7dec6811c1d96b62f3a9d7b834bef41758df021a32c6f1e4dc9059099272f9"} Oct 31 00:49:17.995290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:17.995255 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k" event={"ID":"6a11f261-4cc7-4ca9-8b5f-8920883108b4","Type":"ContainerDied","Data":"046eba82f2e738d8e1b6e122515eb314b0305f5d2f384654466f146218eab092"} Oct 31 00:49:18.007409 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.007142 4519 scope.go:122] "RemoveContainer" containerID="a0e48461a68564f3f5616b4e84451907a7dca742aaa5a86a9e9b91ad867ad0d8" Oct 31 00:49:18.007925 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:18.007894 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e48461a68564f3f5616b4e84451907a7dca742aaa5a86a9e9b91ad867ad0d8\": container with ID starting with a0e48461a68564f3f5616b4e84451907a7dca742aaa5a86a9e9b91ad867ad0d8 not found: ID does not exist" containerID="a0e48461a68564f3f5616b4e84451907a7dca742aaa5a86a9e9b91ad867ad0d8" Oct 31 00:49:18.008038 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.007935 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e48461a68564f3f5616b4e84451907a7dca742aaa5a86a9e9b91ad867ad0d8"} err="failed to get container status \"a0e48461a68564f3f5616b4e84451907a7dca742aaa5a86a9e9b91ad867ad0d8\": rpc error: code = NotFound desc = could not find container \"a0e48461a68564f3f5616b4e84451907a7dca742aaa5a86a9e9b91ad867ad0d8\": container with ID starting with a0e48461a68564f3f5616b4e84451907a7dca742aaa5a86a9e9b91ad867ad0d8 not found: ID does not exist" Oct 31 00:49:18.008038 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.007960 4519 scope.go:122] "RemoveContainer" containerID="9fed18f5c0ad267bdf0f1cd44b9a7f985195622529b61a5fefff5b6c56dccff0" Oct 31 00:49:18.019678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.019626 4519 scope.go:122] "RemoveContainer" containerID="9fed18f5c0ad267bdf0f1cd44b9a7f985195622529b61a5fefff5b6c56dccff0" Oct 31 00:49:18.019960 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:18.019941 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fed18f5c0ad267bdf0f1cd44b9a7f985195622529b61a5fefff5b6c56dccff0\": container with ID starting with 9fed18f5c0ad267bdf0f1cd44b9a7f985195622529b61a5fefff5b6c56dccff0 not found: ID does not exist" containerID="9fed18f5c0ad267bdf0f1cd44b9a7f985195622529b61a5fefff5b6c56dccff0" Oct 31 00:49:18.020096 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.019966 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fed18f5c0ad267bdf0f1cd44b9a7f985195622529b61a5fefff5b6c56dccff0"} err="failed to get container status \"9fed18f5c0ad267bdf0f1cd44b9a7f985195622529b61a5fefff5b6c56dccff0\": rpc error: code = NotFound desc = could not find container \"9fed18f5c0ad267bdf0f1cd44b9a7f985195622529b61a5fefff5b6c56dccff0\": container with ID starting with 9fed18f5c0ad267bdf0f1cd44b9a7f985195622529b61a5fefff5b6c56dccff0 not found: ID does not exist" Oct 31 00:49:18.020096 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.019982 4519 scope.go:122] "RemoveContainer" containerID="0a7dec6811c1d96b62f3a9d7b834bef41758df021a32c6f1e4dc9059099272f9" Oct 31 00:49:18.028918 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.028895 4519 scope.go:122] "RemoveContainer" containerID="0a7dec6811c1d96b62f3a9d7b834bef41758df021a32c6f1e4dc9059099272f9" Oct 31 00:49:18.029263 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:18.029242 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a7dec6811c1d96b62f3a9d7b834bef41758df021a32c6f1e4dc9059099272f9\": container with ID starting with 0a7dec6811c1d96b62f3a9d7b834bef41758df021a32c6f1e4dc9059099272f9 not found: ID does not exist" containerID="0a7dec6811c1d96b62f3a9d7b834bef41758df021a32c6f1e4dc9059099272f9" Oct 31 00:49:18.029358 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.029268 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a7dec6811c1d96b62f3a9d7b834bef41758df021a32c6f1e4dc9059099272f9"} err="failed to get container status \"0a7dec6811c1d96b62f3a9d7b834bef41758df021a32c6f1e4dc9059099272f9\": rpc error: code = NotFound desc = could not find container \"0a7dec6811c1d96b62f3a9d7b834bef41758df021a32c6f1e4dc9059099272f9\": container with ID starting with 0a7dec6811c1d96b62f3a9d7b834bef41758df021a32c6f1e4dc9059099272f9 not found: ID does not exist" Oct 31 00:49:18.053662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.046088 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6"] Oct 31 00:49:18.053662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.046345 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" podUID="cbd4f0b9-e235-40f7-acef-68e1051fc918" containerName="istio-proxy" containerID="cri-o://bb022f8a4c6192c012196d3238742fb91f603287c22564bbff87399536c5ef29" gracePeriod=30 Oct 31 00:49:18.053662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.050971 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng"] Oct 31 00:49:18.053662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.051249 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" containerID="cri-o://bce6075ae55a7ca1f200f28d937d6f827e2581897165e8cfdfe357b770986ef2" gracePeriod=30 Oct 31 00:49:18.064842 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.064806 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn"] Oct 31 00:49:18.110802 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.110771 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-tm2gn"] Oct 31 00:49:18.146856 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.146825 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9"] Oct 31 00:49:18.161383 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.161312 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-l4lq9"] Oct 31 00:49:18.226088 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.226051 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k"] Oct 31 00:49:18.226330 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:18.226314 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-5c2a03-openshift-default-686489c6b5-hz89k"] Oct 31 00:49:19.084197 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:19.084155 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:19.189114 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:19.189085 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:19.286546 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:19.286491 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:19.389358 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:19.389255 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:19.506278 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:19.506247 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:20.100830 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:20.100798 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" path="/var/lib/kubelet/pods/6a11f261-4cc7-4ca9-8b5f-8920883108b4/volumes" Oct 31 00:49:20.101249 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:20.101236 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" path="/var/lib/kubelet/pods/b22ec283-637b-4dd0-add9-76777f64e77c/volumes" Oct 31 00:49:20.101620 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:20.101609 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fddc64cf-c911-4921-b176-a7c119cdd118" path="/var/lib/kubelet/pods/fddc64cf-c911-4921-b176-a7c119cdd118/volumes" Oct 31 00:49:20.667361 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:20.667325 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:22.412386 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:22.412346 4519 patch_prober.go:29] interesting pod/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:49:22.412869 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:22.412408 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" podUID="cbd4f0b9-e235-40f7-acef-68e1051fc918" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:49:23.372774 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.372748 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:49:23.478506 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.478215 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:49:23.486253 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.486233 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:49:23.489682 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.489664 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:49:23.561440 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561403 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a619a89-f559-4918-bbbb-2cf358157e77-istio-token\" (UniqueName: \"kubernetes.io/projected/8a619a89-f559-4918-bbbb-2cf358157e77-istio-token\") pod \"8a619a89-f559-4918-bbbb-2cf358157e77\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " Oct 31 00:49:23.561440 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561451 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-credential-socket\") pod \"8a619a89-f559-4918-bbbb-2cf358157e77\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " Oct 31 00:49:23.561755 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561489 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-workload-certs\") pod \"cbd4f0b9-e235-40f7-acef-68e1051fc918\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " Oct 31 00:49:23.561755 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561520 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-credential-socket\") pod \"9e7a3b29-178b-43e4-b127-314c2b183d9e\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " Oct 31 00:49:23.561755 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561550 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-data\") pod \"9e7a3b29-178b-43e4-b127-314c2b183d9e\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " Oct 31 00:49:23.561755 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561586 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-workload-socket\") pod \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " Oct 31 00:49:23.561755 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561614 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-envoy\") pod \"9e7a3b29-178b-43e4-b127-314c2b183d9e\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " Oct 31 00:49:23.561755 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561667 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-workload-certs\") pod \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " Oct 31 00:49:23.561755 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561694 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-credential-socket\") pod \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " Oct 31 00:49:23.561755 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561723 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-workload-socket\") pod \"8a619a89-f559-4918-bbbb-2cf358157e77\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " Oct 31 00:49:23.562215 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561759 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-kube-api-access-c7gvv\" (UniqueName: \"kubernetes.io/projected/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-kube-api-access-c7gvv\") pod \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " Oct 31 00:49:23.562215 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561790 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-podinfo\") pod \"cbd4f0b9-e235-40f7-acef-68e1051fc918\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " Oct 31 00:49:23.562215 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561816 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e7a3b29-178b-43e4-b127-314c2b183d9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7a3b29-178b-43e4-b127-314c2b183d9e-istiod-ca-cert\") pod \"9e7a3b29-178b-43e4-b127-314c2b183d9e\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " Oct 31 00:49:23.562215 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561841 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-podinfo\") pod \"9e7a3b29-178b-43e4-b127-314c2b183d9e\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " Oct 31 00:49:23.562215 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561876 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-workload-socket\") pod \"cbd4f0b9-e235-40f7-acef-68e1051fc918\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " Oct 31 00:49:23.562215 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561903 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istiod-ca-cert\") pod \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " Oct 31 00:49:23.562215 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561948 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-data\") pod \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " Oct 31 00:49:23.562215 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561973 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-credential-socket\") pod \"cbd4f0b9-e235-40f7-acef-68e1051fc918\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " Oct 31 00:49:23.562215 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.561999 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-workload-certs\") pod \"9e7a3b29-178b-43e4-b127-314c2b183d9e\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " Oct 31 00:49:23.562215 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.562028 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-istio-envoy\") pod \"8a619a89-f559-4918-bbbb-2cf358157e77\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " Oct 31 00:49:23.562215 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.562060 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7a3b29-178b-43e4-b127-314c2b183d9e-kube-api-access-xpnx5\" (UniqueName: \"kubernetes.io/projected/9e7a3b29-178b-43e4-b127-314c2b183d9e-kube-api-access-xpnx5\") pod \"9e7a3b29-178b-43e4-b127-314c2b183d9e\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " Oct 31 00:49:23.562938 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.562340 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-credential-socket" pod "9e7a3b29-178b-43e4-b127-314c2b183d9e" (UID: "9e7a3b29-178b-43e4-b127-314c2b183d9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.562938 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.562676 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-data" pod "9e7a3b29-178b-43e4-b127-314c2b183d9e" (UID: "9e7a3b29-178b-43e4-b127-314c2b183d9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.562938 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.562756 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-credential-socket" pod "8a619a89-f559-4918-bbbb-2cf358157e77" (UID: "8a619a89-f559-4918-bbbb-2cf358157e77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.563115 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.562935 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-credential-socket" pod "cbd4f0b9-e235-40f7-acef-68e1051fc918" (UID: "cbd4f0b9-e235-40f7-acef-68e1051fc918"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.563176 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.563154 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-workload-certs" pod "cbd4f0b9-e235-40f7-acef-68e1051fc918" (UID: "cbd4f0b9-e235-40f7-acef-68e1051fc918"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.563855 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.563328 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-workload-certs" pod "9e7a3b29-178b-43e4-b127-314c2b183d9e" (UID: "9e7a3b29-178b-43e4-b127-314c2b183d9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.563855 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.563575 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-workload-socket" pod "8a619a89-f559-4918-bbbb-2cf358157e77" (UID: "8a619a89-f559-4918-bbbb-2cf358157e77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.563855 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.563707 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-data" pod "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" (UID: "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.563855 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.563745 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-workload-socket" pod "cbd4f0b9-e235-40f7-acef-68e1051fc918" (UID: "cbd4f0b9-e235-40f7-acef-68e1051fc918"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.563855 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.563759 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-token\" (UniqueName: \"kubernetes.io/projected/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-token\") pod \"9e7a3b29-178b-43e4-b127-314c2b183d9e\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " Oct 31 00:49:23.563855 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.563804 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a619a89-f559-4918-bbbb-2cf358157e77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a619a89-f559-4918-bbbb-2cf358157e77-istio-podinfo\") pod \"8a619a89-f559-4918-bbbb-2cf358157e77\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " Oct 31 00:49:23.564709 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.563822 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istiod-ca-cert" pod "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" (UID: "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:23.564817 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.563852 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-envoy\") pod \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " Oct 31 00:49:23.564817 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.564764 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-token\" (UniqueName: \"kubernetes.io/projected/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-token\") pod \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " Oct 31 00:49:23.564947 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.564815 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a619a89-f559-4918-bbbb-2cf358157e77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a619a89-f559-4918-bbbb-2cf358157e77-istiod-ca-cert\") pod \"8a619a89-f559-4918-bbbb-2cf358157e77\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " Oct 31 00:49:23.564947 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.564849 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-workload-socket\") pod \"9e7a3b29-178b-43e4-b127-314c2b183d9e\" (UID: \"9e7a3b29-178b-43e4-b127-314c2b183d9e\") " Oct 31 00:49:23.564947 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.564895 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a619a89-f559-4918-bbbb-2cf358157e77-kube-api-access-mstdx\" (UniqueName: \"kubernetes.io/projected/8a619a89-f559-4918-bbbb-2cf358157e77-kube-api-access-mstdx\") pod \"8a619a89-f559-4918-bbbb-2cf358157e77\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " Oct 31 00:49:23.564947 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.564932 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbd4f0b9-e235-40f7-acef-68e1051fc918-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd4f0b9-e235-40f7-acef-68e1051fc918-istiod-ca-cert\") pod \"cbd4f0b9-e235-40f7-acef-68e1051fc918\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " Oct 31 00:49:23.565160 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.564966 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-workload-certs\") pod \"8a619a89-f559-4918-bbbb-2cf358157e77\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " Oct 31 00:49:23.565160 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.564995 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-istio-data\") pod \"8a619a89-f559-4918-bbbb-2cf358157e77\" (UID: \"8a619a89-f559-4918-bbbb-2cf358157e77\") " Oct 31 00:49:23.565160 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.565025 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-data\") pod \"cbd4f0b9-e235-40f7-acef-68e1051fc918\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " Oct 31 00:49:23.565160 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.565076 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-envoy\") pod \"cbd4f0b9-e235-40f7-acef-68e1051fc918\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " Oct 31 00:49:23.565160 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.565132 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd4f0b9-e235-40f7-acef-68e1051fc918-kube-api-access-rxmkl\" (UniqueName: \"kubernetes.io/projected/cbd4f0b9-e235-40f7-acef-68e1051fc918-kube-api-access-rxmkl\") pod \"cbd4f0b9-e235-40f7-acef-68e1051fc918\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " Oct 31 00:49:23.565438 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.565161 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-token\" (UniqueName: \"kubernetes.io/projected/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-token\") pod \"cbd4f0b9-e235-40f7-acef-68e1051fc918\" (UID: \"cbd4f0b9-e235-40f7-acef-68e1051fc918\") " Oct 31 00:49:23.565438 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.565215 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-podinfo\") pod \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\" (UID: \"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4\") " Oct 31 00:49:23.565781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.565590 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.565781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.565617 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.565781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.565619 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-workload-certs" pod "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" (UID: "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.565781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.565650 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.565781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.565668 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.565781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.565696 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.565781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.565714 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.565781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.565729 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.565781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.565729 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a619a89-f559-4918-bbbb-2cf358157e77-istio-token" pod "8a619a89-f559-4918-bbbb-2cf358157e77" (UID: "8a619a89-f559-4918-bbbb-2cf358157e77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:23.565781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.565744 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.565781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.565759 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.565781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.565773 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.566799 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.566770 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-podinfo" pod "9e7a3b29-178b-43e4-b127-314c2b183d9e" (UID: "9e7a3b29-178b-43e4-b127-314c2b183d9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:23.566911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.566892 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-podinfo" pod "cbd4f0b9-e235-40f7-acef-68e1051fc918" (UID: "cbd4f0b9-e235-40f7-acef-68e1051fc918"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:23.567550 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.567275 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-envoy" pod "9e7a3b29-178b-43e4-b127-314c2b183d9e" (UID: "9e7a3b29-178b-43e4-b127-314c2b183d9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.567550 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.567458 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd4f0b9-e235-40f7-acef-68e1051fc918-istiod-ca-cert" pod "cbd4f0b9-e235-40f7-acef-68e1051fc918" (UID: "cbd4f0b9-e235-40f7-acef-68e1051fc918"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:23.567839 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.567813 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-data" pod "cbd4f0b9-e235-40f7-acef-68e1051fc918" (UID: "cbd4f0b9-e235-40f7-acef-68e1051fc918"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.568288 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.568262 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-workload-socket" pod "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" (UID: "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.568924 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.568885 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-istio-envoy" pod "8a619a89-f559-4918-bbbb-2cf358157e77" (UID: "8a619a89-f559-4918-bbbb-2cf358157e77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.569137 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.569111 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-credential-socket" pod "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" (UID: "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.570071 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.570035 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a619a89-f559-4918-bbbb-2cf358157e77-istiod-ca-cert" pod "8a619a89-f559-4918-bbbb-2cf358157e77" (UID: "8a619a89-f559-4918-bbbb-2cf358157e77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:23.570246 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.570216 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-workload-certs" pod "8a619a89-f559-4918-bbbb-2cf358157e77" (UID: "8a619a89-f559-4918-bbbb-2cf358157e77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.570658 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.570611 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-istio-data" pod "8a619a89-f559-4918-bbbb-2cf358157e77" (UID: "8a619a89-f559-4918-bbbb-2cf358157e77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.571399 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.571340 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-token" pod "cbd4f0b9-e235-40f7-acef-68e1051fc918" (UID: "cbd4f0b9-e235-40f7-acef-68e1051fc918"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:23.571499 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.571457 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-workload-socket" pod "9e7a3b29-178b-43e4-b127-314c2b183d9e" (UID: "9e7a3b29-178b-43e4-b127-314c2b183d9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.571842 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.571784 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7a3b29-178b-43e4-b127-314c2b183d9e-istiod-ca-cert" pod "9e7a3b29-178b-43e4-b127-314c2b183d9e" (UID: "9e7a3b29-178b-43e4-b127-314c2b183d9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:23.572254 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.572218 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-envoy" pod "cbd4f0b9-e235-40f7-acef-68e1051fc918" (UID: "cbd4f0b9-e235-40f7-acef-68e1051fc918"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.572982 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.572948 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a619a89-f559-4918-bbbb-2cf358157e77-istio-podinfo" pod "8a619a89-f559-4918-bbbb-2cf358157e77" (UID: "8a619a89-f559-4918-bbbb-2cf358157e77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:23.574206 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.574170 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-token" pod "9e7a3b29-178b-43e4-b127-314c2b183d9e" (UID: "9e7a3b29-178b-43e4-b127-314c2b183d9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:23.574480 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.574459 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7a3b29-178b-43e4-b127-314c2b183d9e-kube-api-access-xpnx5" pod "9e7a3b29-178b-43e4-b127-314c2b183d9e" (UID: "9e7a3b29-178b-43e4-b127-314c2b183d9e"). InnerVolumeSpecName "kube-api-access-xpnx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:23.575046 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.574927 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-podinfo" pod "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" (UID: "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:23.575046 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.574942 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd4f0b9-e235-40f7-acef-68e1051fc918-kube-api-access-rxmkl" pod "cbd4f0b9-e235-40f7-acef-68e1051fc918" (UID: "cbd4f0b9-e235-40f7-acef-68e1051fc918"). InnerVolumeSpecName "kube-api-access-rxmkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:23.575263 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.575068 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-kube-api-access-c7gvv" pod "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" (UID: "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4"). InnerVolumeSpecName "kube-api-access-c7gvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:23.575263 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.575085 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-envoy" pod "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" (UID: "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:23.575535 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.575477 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a619a89-f559-4918-bbbb-2cf358157e77-kube-api-access-mstdx" pod "8a619a89-f559-4918-bbbb-2cf358157e77" (UID: "8a619a89-f559-4918-bbbb-2cf358157e77"). InnerVolumeSpecName "kube-api-access-mstdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:23.576171 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.576147 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-token" pod "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" (UID: "d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:23.667025 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.666943 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mstdx\" (UniqueName: \"kubernetes.io/projected/8a619a89-f559-4918-bbbb-2cf358157e77-kube-api-access-mstdx\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667025 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.666984 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd4f0b9-e235-40f7-acef-68e1051fc918-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667025 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667004 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667025 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667019 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667034 4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-data\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667048 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667064 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxmkl\" (UniqueName: \"kubernetes.io/projected/cbd4f0b9-e235-40f7-acef-68e1051fc918-kube-api-access-rxmkl\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667078 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667092 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667106 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a619a89-f559-4918-bbbb-2cf358157e77-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667120 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667135 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667148 4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-workload-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667161 4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-credential-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667174 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7gvv\" (UniqueName: \"kubernetes.io/projected/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-kube-api-access-c7gvv\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667197 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd4f0b9-e235-40f7-acef-68e1051fc918-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667222 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7a3b29-178b-43e4-b127-314c2b183d9e-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667236 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667250 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a619a89-f559-4918-bbbb-2cf358157e77-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667263 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpnx5\" (UniqueName: \"kubernetes.io/projected/9e7a3b29-178b-43e4-b127-314c2b183d9e-kube-api-access-xpnx5\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667276 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e7a3b29-178b-43e4-b127-314c2b183d9e-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667289 4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a619a89-f559-4918-bbbb-2cf358157e77-istio-podinfo\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667848 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667305 4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-envoy\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667848 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667320 4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4-istio-token\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667848 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667334 4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a619a89-f559-4918-bbbb-2cf358157e77-istiod-ca-cert\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:23.667848 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:23.667347 4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7a3b29-178b-43e4-b127-314c2b183d9e-workload-socket\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:24.077664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.077214 4519 generic.go:356] "Generic (PLEG): container finished" podID="cbd4f0b9-e235-40f7-acef-68e1051fc918" containerID="bb022f8a4c6192c012196d3238742fb91f603287c22564bbff87399536c5ef29" exitCode=0 Oct 31 00:49:24.077664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.077414 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" event={"ID":"cbd4f0b9-e235-40f7-acef-68e1051fc918","Type":"ContainerDied","Data":"bb022f8a4c6192c012196d3238742fb91f603287c22564bbff87399536c5ef29"} Oct 31 00:49:24.077664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.077469 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" event={"ID":"cbd4f0b9-e235-40f7-acef-68e1051fc918","Type":"ContainerDied","Data":"5c622d48770f27b2e181435fad3b6b89f9e8f130f904de541d63383cfe4f2f01"} Oct 31 00:49:24.077664 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.077504 4519 scope.go:122] "RemoveContainer" containerID="bb022f8a4c6192c012196d3238742fb91f603287c22564bbff87399536c5ef29" Oct 31 00:49:24.105612 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.078073 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6" Oct 31 00:49:24.105612 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.092924 4519 generic.go:356] "Generic (PLEG): container finished" podID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerID="9e33b8afb51f75baebf9b8dc648d0c80d472fd199f646373c4dfff580adb7b54" exitCode=0 Oct 31 00:49:24.105612 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.093035 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" event={"ID":"9e7a3b29-178b-43e4-b127-314c2b183d9e","Type":"ContainerDied","Data":"9e33b8afb51f75baebf9b8dc648d0c80d472fd199f646373c4dfff580adb7b54"} Oct 31 00:49:24.105612 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.093082 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" event={"ID":"9e7a3b29-178b-43e4-b127-314c2b183d9e","Type":"ContainerDied","Data":"363b3e257dbdad72860ea518e1517a2f1958b2d68128f7d2fc68e7944254e04a"} Oct 31 00:49:24.105612 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.093154 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d" Oct 31 00:49:24.105612 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.103483 4519 generic.go:356] "Generic (PLEG): container finished" podID="8a619a89-f559-4918-bbbb-2cf358157e77" containerID="fce4eb71bd9706f20425a86690448aa91797027e9680f4f323fb185d698fc2e4" exitCode=0 Oct 31 00:49:24.105612 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.103621 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" Oct 31 00:49:24.106737 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.106711 4519 scope.go:122] "RemoveContainer" containerID="bb022f8a4c6192c012196d3238742fb91f603287c22564bbff87399536c5ef29" Oct 31 00:49:24.107194 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:24.107162 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb022f8a4c6192c012196d3238742fb91f603287c22564bbff87399536c5ef29\": container with ID starting with bb022f8a4c6192c012196d3238742fb91f603287c22564bbff87399536c5ef29 not found: ID does not exist" containerID="bb022f8a4c6192c012196d3238742fb91f603287c22564bbff87399536c5ef29" Oct 31 00:49:24.107281 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.107205 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb022f8a4c6192c012196d3238742fb91f603287c22564bbff87399536c5ef29"} err="failed to get container status \"bb022f8a4c6192c012196d3238742fb91f603287c22564bbff87399536c5ef29\": rpc error: code = NotFound desc = could not find container \"bb022f8a4c6192c012196d3238742fb91f603287c22564bbff87399536c5ef29\": container with ID starting with bb022f8a4c6192c012196d3238742fb91f603287c22564bbff87399536c5ef29 not found: ID does not exist" Oct 31 00:49:24.107281 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.107231 4519 scope.go:122] "RemoveContainer" containerID="9e33b8afb51f75baebf9b8dc648d0c80d472fd199f646373c4dfff580adb7b54" Oct 31 00:49:24.108827 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.108560 4519 generic.go:356] "Generic (PLEG): container finished" podID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerID="bce6075ae55a7ca1f200f28d937d6f827e2581897165e8cfdfe357b770986ef2" exitCode=0 Oct 31 00:49:24.108827 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.108654 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" Oct 31 00:49:24.112598 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.110775 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" event={"ID":"8a619a89-f559-4918-bbbb-2cf358157e77","Type":"ContainerDied","Data":"fce4eb71bd9706f20425a86690448aa91797027e9680f4f323fb185d698fc2e4"} Oct 31 00:49:24.112598 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.110834 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm" event={"ID":"8a619a89-f559-4918-bbbb-2cf358157e77","Type":"ContainerDied","Data":"e8719fd195b7e49cd9de8bee1a91870de5ec13876861944e20ae4dc30513c641"} Oct 31 00:49:24.112598 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.110859 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" event={"ID":"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4","Type":"ContainerDied","Data":"bce6075ae55a7ca1f200f28d937d6f827e2581897165e8cfdfe357b770986ef2"} Oct 31 00:49:24.112598 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.110884 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng" event={"ID":"d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4","Type":"ContainerDied","Data":"f94f6948c3a43a81b27cac0bd1f3578c2252cd22986b0a0c616a0ce477ba82e4"} Oct 31 00:49:24.209770 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.209737 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d"] Oct 31 00:49:24.209919 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.209892 4519 scope.go:122] "RemoveContainer" containerID="9e33b8afb51f75baebf9b8dc648d0c80d472fd199f646373c4dfff580adb7b54" Oct 31 00:49:24.210313 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:24.210279 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e33b8afb51f75baebf9b8dc648d0c80d472fd199f646373c4dfff580adb7b54\": container with ID starting with 9e33b8afb51f75baebf9b8dc648d0c80d472fd199f646373c4dfff580adb7b54 not found: ID does not exist" containerID="9e33b8afb51f75baebf9b8dc648d0c80d472fd199f646373c4dfff580adb7b54" Oct 31 00:49:24.210439 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.210323 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e33b8afb51f75baebf9b8dc648d0c80d472fd199f646373c4dfff580adb7b54"} err="failed to get container status \"9e33b8afb51f75baebf9b8dc648d0c80d472fd199f646373c4dfff580adb7b54\": rpc error: code = NotFound desc = could not find container \"9e33b8afb51f75baebf9b8dc648d0c80d472fd199f646373c4dfff580adb7b54\": container with ID starting with 9e33b8afb51f75baebf9b8dc648d0c80d472fd199f646373c4dfff580adb7b54 not found: ID does not exist" Oct 31 00:49:24.210439 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.210346 4519 scope.go:122] "RemoveContainer" containerID="fce4eb71bd9706f20425a86690448aa91797027e9680f4f323fb185d698fc2e4" Oct 31 00:49:24.212604 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.212528 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-8kf6d"] Oct 31 00:49:24.220889 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.220870 4519 scope.go:122] "RemoveContainer" containerID="fce4eb71bd9706f20425a86690448aa91797027e9680f4f323fb185d698fc2e4" Oct 31 00:49:24.221172 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:24.221149 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce4eb71bd9706f20425a86690448aa91797027e9680f4f323fb185d698fc2e4\": container with ID starting with fce4eb71bd9706f20425a86690448aa91797027e9680f4f323fb185d698fc2e4 not found: ID does not exist" containerID="fce4eb71bd9706f20425a86690448aa91797027e9680f4f323fb185d698fc2e4" Oct 31 00:49:24.221225 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.221179 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce4eb71bd9706f20425a86690448aa91797027e9680f4f323fb185d698fc2e4"} err="failed to get container status \"fce4eb71bd9706f20425a86690448aa91797027e9680f4f323fb185d698fc2e4\": rpc error: code = NotFound desc = could not find container \"fce4eb71bd9706f20425a86690448aa91797027e9680f4f323fb185d698fc2e4\": container with ID starting with fce4eb71bd9706f20425a86690448aa91797027e9680f4f323fb185d698fc2e4 not found: ID does not exist" Oct 31 00:49:24.221225 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.221194 4519 scope.go:122] "RemoveContainer" containerID="bce6075ae55a7ca1f200f28d937d6f827e2581897165e8cfdfe357b770986ef2" Oct 31 00:49:24.270711 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.225176 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6"] Oct 31 00:49:24.301774 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.301697 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-9scf6"] Oct 31 00:49:24.306411 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.306383 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng"] Oct 31 00:49:24.307370 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.307350 4519 scope.go:122] "RemoveContainer" containerID="bce6075ae55a7ca1f200f28d937d6f827e2581897165e8cfdfe357b770986ef2" Oct 31 00:49:24.307892 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:24.307864 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce6075ae55a7ca1f200f28d937d6f827e2581897165e8cfdfe357b770986ef2\": container with ID starting with bce6075ae55a7ca1f200f28d937d6f827e2581897165e8cfdfe357b770986ef2 not found: ID does not exist" containerID="bce6075ae55a7ca1f200f28d937d6f827e2581897165e8cfdfe357b770986ef2" Oct 31 00:49:24.307979 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.307902 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce6075ae55a7ca1f200f28d937d6f827e2581897165e8cfdfe357b770986ef2"} err="failed to get container status \"bce6075ae55a7ca1f200f28d937d6f827e2581897165e8cfdfe357b770986ef2\": rpc error: code = NotFound desc = could not find container \"bce6075ae55a7ca1f200f28d937d6f827e2581897165e8cfdfe357b770986ef2\": container with ID starting with bce6075ae55a7ca1f200f28d937d6f827e2581897165e8cfdfe357b770986ef2 not found: ID does not exist" Oct 31 00:49:24.307979 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.307947 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-nlkng"] Oct 31 00:49:24.309549 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.309531 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm"] Oct 31 00:49:24.310713 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:24.310695 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b433e6-openshift-default-755cb85cc5-wpftm"] Oct 31 00:49:25.109833 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:25.109793 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ggdf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:25.110334 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:25.109853 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" podUID="1a6c37d5-3025-470f-94c0-ee1502fc7a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:26.101911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:26.101836 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" path="/var/lib/kubelet/pods/8a619a89-f559-4918-bbbb-2cf358157e77/volumes" Oct 31 00:49:26.102286 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:26.102273 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" path="/var/lib/kubelet/pods/9e7a3b29-178b-43e4-b127-314c2b183d9e/volumes" Oct 31 00:49:26.102674 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:26.102662 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbd4f0b9-e235-40f7-acef-68e1051fc918" path="/var/lib/kubelet/pods/cbd4f0b9-e235-40f7-acef-68e1051fc918/volumes" Oct 31 00:49:26.103358 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:26.103344 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" path="/var/lib/kubelet/pods/d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4/volumes" Oct 31 00:49:26.287511 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:26.287471 4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a619a89_f559_4918_bbbb_2cf358157e77.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:49:26.811332 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:26.811299 4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a619a89_f559_4918_bbbb_2cf358157e77.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:49:26.811555 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:26.811312 4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a619a89_f559_4918_bbbb_2cf358157e77.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:49:27.150388 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:27.150295 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:27.241562 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:27.241524 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:27.341330 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:27.341294 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:27.710553 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:27.710520 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:32.941724 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:32.941685 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:34.037444 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:34.037404 4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a619a89_f559_4918_bbbb_2cf358157e77.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:49:41.291838 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.291800 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2"] Oct 31 00:49:41.292309 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292273 4519 cpu_manager.go:414] "containerMap: removing container" podUID="6b03043a-75b4-404a-800f-d71f33a04108" containerName="payload-processing" Oct 31 00:49:41.292309 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292289 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b03043a-75b4-404a-800f-d71f33a04108" containerName="payload-processing" Oct 31 00:49:41.292384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292309 4519 cpu_manager.go:414] "containerMap: removing container" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" Oct 31 00:49:41.292384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292319 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" Oct 31 00:49:41.292384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292330 4519 cpu_manager.go:414] "containerMap: removing container" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" Oct 31 00:49:41.292384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292339 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" Oct 31 00:49:41.292384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292354 4519 cpu_manager.go:414] "containerMap: removing container" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" Oct 31 00:49:41.292384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292361 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" Oct 31 00:49:41.292384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292374 4519 cpu_manager.go:414] "containerMap: removing container" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" Oct 31 00:49:41.292384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292382 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292393 4519 cpu_manager.go:414] "containerMap: removing container" podUID="cbd4f0b9-e235-40f7-acef-68e1051fc918" containerName="istio-proxy" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292401 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd4f0b9-e235-40f7-acef-68e1051fc918" containerName="istio-proxy" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292411 4519 cpu_manager.go:414] "containerMap: removing container" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292418 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292428 4519 cpu_manager.go:414] "containerMap: removing container" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292437 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292447 4519 cpu_manager.go:414] "containerMap: removing container" podUID="234959cb-ee8c-4761-9032-ec19e95d9106" containerName="revoke-keys" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292455 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="234959cb-ee8c-4761-9032-ec19e95d9106" containerName="revoke-keys" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292471 4519 cpu_manager.go:414] "containerMap: removing container" podUID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" containerName="istio-proxy" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292478 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" containerName="istio-proxy" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292493 4519 cpu_manager.go:414] "containerMap: removing container" podUID="cdbf7b30-65d4-4462-af10-05cdd5015ce3" containerName="istio-proxy" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292499 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdbf7b30-65d4-4462-af10-05cdd5015ce3" containerName="istio-proxy" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292516 4519 cpu_manager.go:414] "containerMap: removing container" podUID="f76bbc2e-d9b0-4ac4-a09d-91b609760c16" containerName="payload-pre-processing" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292523 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76bbc2e-d9b0-4ac4-a09d-91b609760c16" containerName="payload-pre-processing" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292534 4519 cpu_manager.go:414] "containerMap: removing container" podUID="fddc64cf-c911-4921-b176-a7c119cdd118" containerName="istio-proxy" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292542 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fddc64cf-c911-4921-b176-a7c119cdd118" containerName="istio-proxy" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292551 4519 cpu_manager.go:414] "containerMap: removing container" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292559 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292574 4519 cpu_manager.go:414] "containerMap: removing container" podUID="9c1e90c3-947d-41f2-a4bb-fca7f4b54c40" containerName="maas-api" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292582 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1e90c3-947d-41f2-a4bb-fca7f4b54c40" containerName="maas-api" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292599 4519 cpu_manager.go:414] "containerMap: removing container" podUID="4edab197-6a3f-457a-acab-82d4cebc7b0f" containerName="istio-proxy" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292607 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4edab197-6a3f-457a-acab-82d4cebc7b0f" containerName="istio-proxy" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292624 4519 cpu_manager.go:414] "containerMap: removing container" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292647 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" Oct 31 00:49:41.292652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292659 4519 cpu_manager.go:414] "containerMap: removing container" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292667 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292752 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76bbc2e-d9b0-4ac4-a09d-91b609760c16" containerName="payload-pre-processing" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292764 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="fddc64cf-c911-4921-b176-a7c119cdd118" containerName="istio-proxy" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292773 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a619a89-f559-4918-bbbb-2cf358157e77" containerName="istio-proxy" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292782 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1e90c3-947d-41f2-a4bb-fca7f4b54c40" containerName="maas-api" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292793 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4edab197-6a3f-457a-acab-82d4cebc7b0f" containerName="istio-proxy" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292804 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e7a3b29-178b-43e4-b127-314c2b183d9e" containerName="istio-proxy" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292811 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b22ec283-637b-4dd0-add9-76777f64e77c" containerName="istio-proxy" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292820 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b03043a-75b4-404a-800f-d71f33a04108" containerName="payload-processing" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292831 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d9d315d-8875-4e45-8d34-2005ddd628d5" containerName="main" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292841 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5fc25c3-483d-4644-b774-815f60af2025" containerName="istio-proxy" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292849 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="63751bb9-e9b0-4c48-9887-2bf875a39d87" containerName="main" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292859 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4298cde5-e51e-42f5-924d-0c2224a1f57e" containerName="main" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292867 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd4f0b9-e235-40f7-acef-68e1051fc918" containerName="istio-proxy" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292875 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="193c8acd-7eb4-49f3-b2b3-0c398cdd9c02" containerName="main" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292884 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d69ce0fa-acb7-4e5a-b578-6333d4cf0ae4" containerName="istio-proxy" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292891 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="234959cb-ee8c-4761-9032-ec19e95d9106" containerName="revoke-keys" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292900 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a11f261-4cc7-4ca9-8b5f-8920883108b4" containerName="istio-proxy" Oct 31 00:49:41.293384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.292909 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdbf7b30-65d4-4462-af10-05cdd5015ce3" containerName="istio-proxy" Oct 31 00:49:41.296316 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.296295 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" Oct 31 00:49:41.304736 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.304020 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2"] Oct 31 00:49:41.312548 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.312516 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:49:41.312741 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.312600 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:41.312868 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.312851 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:49:41.313600 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.313578 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:49:41.322708 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:41.322677 4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a619a89_f559_4918_bbbb_2cf358157e77.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:49:41.345742 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.345710 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj99l\" (UniqueName: \"kubernetes.io/projected/e57ee0ab-0cb5-4441-a910-49530df6fdc0-kube-api-access-bj99l\") pod \"maas-api-revoke-sub-85237e45-5ghz2\" (UID: \"e57ee0ab-0cb5-4441-a910-49530df6fdc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" Oct 31 00:49:41.345939 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.345763 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57ee0ab-0cb5-4441-a910-49530df6fdc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-5ghz2\" (UID: \"e57ee0ab-0cb5-4441-a910-49530df6fdc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" Oct 31 00:49:41.345939 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.345789 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57ee0ab-0cb5-4441-a910-49530df6fdc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-5ghz2\" (UID: \"e57ee0ab-0cb5-4441-a910-49530df6fdc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" Oct 31 00:49:41.446845 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.446794 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57ee0ab-0cb5-4441-a910-49530df6fdc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-5ghz2\" (UID: \"e57ee0ab-0cb5-4441-a910-49530df6fdc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" Oct 31 00:49:41.446845 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.446846 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57ee0ab-0cb5-4441-a910-49530df6fdc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-5ghz2\" (UID: \"e57ee0ab-0cb5-4441-a910-49530df6fdc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" Oct 31 00:49:41.447057 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.446912 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj99l\" (UniqueName: \"kubernetes.io/projected/e57ee0ab-0cb5-4441-a910-49530df6fdc0-kube-api-access-bj99l\") pod \"maas-api-revoke-sub-85237e45-5ghz2\" (UID: \"e57ee0ab-0cb5-4441-a910-49530df6fdc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" Oct 31 00:49:41.447548 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.447527 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57ee0ab-0cb5-4441-a910-49530df6fdc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-5ghz2\" (UID: \"e57ee0ab-0cb5-4441-a910-49530df6fdc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" Oct 31 00:49:41.449186 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.449167 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57ee0ab-0cb5-4441-a910-49530df6fdc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-5ghz2\" (UID: \"e57ee0ab-0cb5-4441-a910-49530df6fdc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" Oct 31 00:49:41.456720 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.456694 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj99l\" (UniqueName: \"kubernetes.io/projected/e57ee0ab-0cb5-4441-a910-49530df6fdc0-kube-api-access-bj99l\") pod \"maas-api-revoke-sub-85237e45-5ghz2\" (UID: \"e57ee0ab-0cb5-4441-a910-49530df6fdc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" Oct 31 00:49:41.625375 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.625293 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" Oct 31 00:49:41.896050 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:41.896024 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2"] Oct 31 00:49:41.897704 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:49:41.897675 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57ee0ab_0cb5_4441_a910_49530df6fdc0.slice/crio-53d9dc829974573dbe35bfd8422c78c93350ee756d44e8e4ed9e432a8d5a43c1 WatchSource:0}: Error finding container 53d9dc829974573dbe35bfd8422c78c93350ee756d44e8e4ed9e432a8d5a43c1: Status 404 returned error can't find the container with id 53d9dc829974573dbe35bfd8422c78c93350ee756d44e8e4ed9e432a8d5a43c1 Oct 31 00:49:42.199828 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:42.199794 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" event={"ID":"e57ee0ab-0cb5-4441-a910-49530df6fdc0","Type":"ContainerStarted","Data":"934c4435861a803edaf7a81beb6435f5949519ad2d6c3f47ec3ae78abde56787"} Oct 31 00:49:42.199989 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:42.199844 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" event={"ID":"e57ee0ab-0cb5-4441-a910-49530df6fdc0","Type":"ContainerStarted","Data":"53d9dc829974573dbe35bfd8422c78c93350ee756d44e8e4ed9e432a8d5a43c1"} Oct 31 00:49:42.219989 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:42.219944 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" podStartSLOduration=1.21993019 podStartE2EDuration="1.21993019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:42.217720399 +0000 UTC m=+978.246900803" watchObservedRunningTime="2026-10-31 00:49:42.21993019 +0000 UTC m=+978.249110579" Oct 31 00:49:43.206698 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:43.206657 4519 generic.go:356] "Generic (PLEG): container finished" podID="e57ee0ab-0cb5-4441-a910-49530df6fdc0" containerID="934c4435861a803edaf7a81beb6435f5949519ad2d6c3f47ec3ae78abde56787" exitCode=0 Oct 31 00:49:43.207252 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:43.206744 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" event={"ID":"e57ee0ab-0cb5-4441-a910-49530df6fdc0","Type":"ContainerDied","Data":"934c4435861a803edaf7a81beb6435f5949519ad2d6c3f47ec3ae78abde56787"} Oct 31 00:49:44.049406 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:44.049367 4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a619a89_f559_4918_bbbb_2cf358157e77.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:49:44.376765 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:44.376733 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" Oct 31 00:49:44.475209 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:44.475163 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e57ee0ab-0cb5-4441-a910-49530df6fdc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57ee0ab-0cb5-4441-a910-49530df6fdc0-maas-api-openshift-ca\") pod \"e57ee0ab-0cb5-4441-a910-49530df6fdc0\" (UID: \"e57ee0ab-0cb5-4441-a910-49530df6fdc0\") " Oct 31 00:49:44.475421 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:44.475265 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e57ee0ab-0cb5-4441-a910-49530df6fdc0-kube-api-access-bj99l\" (UniqueName: \"kubernetes.io/projected/e57ee0ab-0cb5-4441-a910-49530df6fdc0-kube-api-access-bj99l\") pod \"e57ee0ab-0cb5-4441-a910-49530df6fdc0\" (UID: \"e57ee0ab-0cb5-4441-a910-49530df6fdc0\") " Oct 31 00:49:44.475421 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:44.475326 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e57ee0ab-0cb5-4441-a910-49530df6fdc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57ee0ab-0cb5-4441-a910-49530df6fdc0-maas-api-xks-ca\") pod \"e57ee0ab-0cb5-4441-a910-49530df6fdc0\" (UID: \"e57ee0ab-0cb5-4441-a910-49530df6fdc0\") " Oct 31 00:49:44.475645 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:44.475609 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57ee0ab-0cb5-4441-a910-49530df6fdc0-maas-api-openshift-ca" pod "e57ee0ab-0cb5-4441-a910-49530df6fdc0" (UID: "e57ee0ab-0cb5-4441-a910-49530df6fdc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:44.477480 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:44.477452 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57ee0ab-0cb5-4441-a910-49530df6fdc0-kube-api-access-bj99l" pod "e57ee0ab-0cb5-4441-a910-49530df6fdc0" (UID: "e57ee0ab-0cb5-4441-a910-49530df6fdc0"). InnerVolumeSpecName "kube-api-access-bj99l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:44.477966 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:44.477943 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e57ee0ab-0cb5-4441-a910-49530df6fdc0-maas-api-xks-ca" pod "e57ee0ab-0cb5-4441-a910-49530df6fdc0" (UID: "e57ee0ab-0cb5-4441-a910-49530df6fdc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:44.577000 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:44.576954 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57ee0ab-0cb5-4441-a910-49530df6fdc0-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:44.577000 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:44.576995 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57ee0ab-0cb5-4441-a910-49530df6fdc0-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:44.577000 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:44.577009 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj99l\" (UniqueName: \"kubernetes.io/projected/e57ee0ab-0cb5-4441-a910-49530df6fdc0-kube-api-access-bj99l\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:45.217555 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:45.217523 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" Oct 31 00:49:45.217761 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:45.217561 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2" event={"ID":"e57ee0ab-0cb5-4441-a910-49530df6fdc0","Type":"ContainerDied","Data":"53d9dc829974573dbe35bfd8422c78c93350ee756d44e8e4ed9e432a8d5a43c1"} Oct 31 00:49:45.217761 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:45.217619 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53d9dc829974573dbe35bfd8422c78c93350ee756d44e8e4ed9e432a8d5a43c1" Oct 31 00:49:47.540830 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:47.540787 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:49:48.599296 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:48.599252 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:48.700410 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:48.700380 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:52.702425 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:52.702383 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:54.089312 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:54.089277 4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a619a89_f559_4918_bbbb_2cf358157e77.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:49:54.504729 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:49:54.504696 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:56.280497 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:49:56.280460 4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a619a89_f559_4918_bbbb_2cf358157e77.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:50:04.128258 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:50:04.128219 4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a619a89_f559_4918_bbbb_2cf358157e77.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:50:10.026486 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:10.026449 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59"] Oct 31 00:50:10.031068 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:10.031041 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-310a80d5-tjk59"] Oct 31 00:50:10.100506 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:10.100473 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="324ec116-8c6f-4983-b1eb-99f935102520" path="/var/lib/kubelet/pods/324ec116-8c6f-4983-b1eb-99f935102520/volumes" Oct 31 00:50:11.302607 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:50:11.302576 4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a619a89_f559_4918_bbbb_2cf358157e77.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:50:11.451606 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:11.451571 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:14.138264 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:50:14.138229 4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a619a89_f559_4918_bbbb_2cf358157e77.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:50:17.540470 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:50:17.540434 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:50:19.021676 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:19.021628 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49"] Oct 31 00:50:19.025587 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:19.025552 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-909ea630-66j49"] Oct 31 00:50:20.027053 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:20.027011 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp"] Oct 31 00:50:20.032739 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:20.032707 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-874cf9d5-kn2rp"] Oct 31 00:50:20.100670 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:20.100618 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c603a9a-871a-40bc-86e7-022941340d89" path="/var/lib/kubelet/pods/0c603a9a-871a-40bc-86e7-022941340d89/volumes" Oct 31 00:50:20.101145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:20.101128 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dfb7609-1a91-4376-90d1-b4baf6047398" path="/var/lib/kubelet/pods/6dfb7609-1a91-4376-90d1-b4baf6047398/volumes" Oct 31 00:50:22.360172 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:22.360131 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:24.177399 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:50:24.177368 4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a619a89_f559_4918_bbbb_2cf358157e77.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:50:27.323761 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:27.323718 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:34.029701 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:34.029658 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l"] Oct 31 00:50:34.031801 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:34.031778 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32426f48-xxf4l"] Oct 31 00:50:34.100677 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:34.100622 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65862e2f-a808-4b28-a903-a1defecc2a75" path="/var/lib/kubelet/pods/65862e2f-a808-4b28-a903-a1defecc2a75/volumes" Oct 31 00:50:35.031865 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:35.031829 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd"] Oct 31 00:50:35.036660 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:35.035273 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3817b7d5-zdxkd"] Oct 31 00:50:36.099969 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:36.099897 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edd43374-f0a4-4e87-ab46-cac2f75281cd" path="/var/lib/kubelet/pods/edd43374-f0a4-4e87-ab46-cac2f75281cd/volumes" Oct 31 00:50:37.111183 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.111145 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw"] Oct 31 00:50:37.111594 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.111464 4519 cpu_manager.go:414] "containerMap: removing container" podUID="e57ee0ab-0cb5-4441-a910-49530df6fdc0" containerName="revoke-keys" Oct 31 00:50:37.111594 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.111474 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e57ee0ab-0cb5-4441-a910-49530df6fdc0" containerName="revoke-keys" Oct 31 00:50:37.111594 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.111538 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e57ee0ab-0cb5-4441-a910-49530df6fdc0" containerName="revoke-keys" Oct 31 00:50:37.115672 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.115652 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" Oct 31 00:50:37.120801 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.120773 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:50:37.120987 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.120877 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:50:37.120987 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.120982 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:50:37.121123 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.121020 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:50:37.121685 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.121657 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw"] Oct 31 00:50:37.149348 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.149308 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e6cf90-2c26-4881-bd77-798d533b7dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tfxbw\" (UID: \"54e6cf90-2c26-4881-bd77-798d533b7dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" Oct 31 00:50:37.149538 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.149373 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smnrb\" (UniqueName: \"kubernetes.io/projected/54e6cf90-2c26-4881-bd77-798d533b7dd3-kube-api-access-smnrb\") pod \"maas-api-revoke-sub-c1e271a9-tfxbw\" (UID: \"54e6cf90-2c26-4881-bd77-798d533b7dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" Oct 31 00:50:37.149624 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.149599 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e6cf90-2c26-4881-bd77-798d533b7dd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tfxbw\" (UID: \"54e6cf90-2c26-4881-bd77-798d533b7dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" Oct 31 00:50:37.250425 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.250388 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smnrb\" (UniqueName: \"kubernetes.io/projected/54e6cf90-2c26-4881-bd77-798d533b7dd3-kube-api-access-smnrb\") pod \"maas-api-revoke-sub-c1e271a9-tfxbw\" (UID: \"54e6cf90-2c26-4881-bd77-798d533b7dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" Oct 31 00:50:37.250611 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.250443 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e6cf90-2c26-4881-bd77-798d533b7dd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tfxbw\" (UID: \"54e6cf90-2c26-4881-bd77-798d533b7dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" Oct 31 00:50:37.250611 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.250500 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e6cf90-2c26-4881-bd77-798d533b7dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tfxbw\" (UID: \"54e6cf90-2c26-4881-bd77-798d533b7dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" Oct 31 00:50:37.251276 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.251249 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e6cf90-2c26-4881-bd77-798d533b7dd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tfxbw\" (UID: \"54e6cf90-2c26-4881-bd77-798d533b7dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" Oct 31 00:50:37.252990 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.252965 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e6cf90-2c26-4881-bd77-798d533b7dd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tfxbw\" (UID: \"54e6cf90-2c26-4881-bd77-798d533b7dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" Oct 31 00:50:37.259094 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.259065 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smnrb\" (UniqueName: \"kubernetes.io/projected/54e6cf90-2c26-4881-bd77-798d533b7dd3-kube-api-access-smnrb\") pod \"maas-api-revoke-sub-c1e271a9-tfxbw\" (UID: \"54e6cf90-2c26-4881-bd77-798d533b7dd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" Oct 31 00:50:37.430352 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.430240 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" Oct 31 00:50:37.739480 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:37.739455 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw"] Oct 31 00:50:37.741506 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:50:37.741475 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e6cf90_2c26_4881_bd77_798d533b7dd3.slice/crio-30eea3b961eec86bf592212747125de96bce1cf21812076d87d06a07137e6917 WatchSource:0}: Error finding container 30eea3b961eec86bf592212747125de96bce1cf21812076d87d06a07137e6917: Status 404 returned error can't find the container with id 30eea3b961eec86bf592212747125de96bce1cf21812076d87d06a07137e6917 Oct 31 00:50:38.485502 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:38.485462 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" event={"ID":"54e6cf90-2c26-4881-bd77-798d533b7dd3","Type":"ContainerStarted","Data":"6d9c2e99cd1081a5ba4d4678858d479a127d7386d59eea74bfa6d4fc1e874e51"} Oct 31 00:50:38.485915 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:38.485528 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" event={"ID":"54e6cf90-2c26-4881-bd77-798d533b7dd3","Type":"ContainerStarted","Data":"30eea3b961eec86bf592212747125de96bce1cf21812076d87d06a07137e6917"} Oct 31 00:50:38.503600 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:38.503555 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" podStartSLOduration=1.503542447 podStartE2EDuration="1.503542447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:38.501967088 +0000 UTC m=+1034.531147479" watchObservedRunningTime="2026-10-31 00:50:38.503542447 +0000 UTC m=+1034.532722836" Oct 31 00:50:39.492103 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:39.492063 4519 generic.go:356] "Generic (PLEG): container finished" podID="54e6cf90-2c26-4881-bd77-798d533b7dd3" containerID="6d9c2e99cd1081a5ba4d4678858d479a127d7386d59eea74bfa6d4fc1e874e51" exitCode=0 Oct 31 00:50:39.492588 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:39.492118 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" event={"ID":"54e6cf90-2c26-4881-bd77-798d533b7dd3","Type":"ContainerDied","Data":"6d9c2e99cd1081a5ba4d4678858d479a127d7386d59eea74bfa6d4fc1e874e51"} Oct 31 00:50:40.631190 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:40.631166 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" Oct 31 00:50:40.682725 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:40.682689 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e6cf90-2c26-4881-bd77-798d533b7dd3-kube-api-access-smnrb\" (UniqueName: \"kubernetes.io/projected/54e6cf90-2c26-4881-bd77-798d533b7dd3-kube-api-access-smnrb\") pod \"54e6cf90-2c26-4881-bd77-798d533b7dd3\" (UID: \"54e6cf90-2c26-4881-bd77-798d533b7dd3\") " Oct 31 00:50:40.682909 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:40.682733 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54e6cf90-2c26-4881-bd77-798d533b7dd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e6cf90-2c26-4881-bd77-798d533b7dd3-maas-api-xks-ca\") pod \"54e6cf90-2c26-4881-bd77-798d533b7dd3\" (UID: \"54e6cf90-2c26-4881-bd77-798d533b7dd3\") " Oct 31 00:50:40.682909 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:40.682787 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e6cf90-2c26-4881-bd77-798d533b7dd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e6cf90-2c26-4881-bd77-798d533b7dd3-maas-api-openshift-ca\") pod \"54e6cf90-2c26-4881-bd77-798d533b7dd3\" (UID: \"54e6cf90-2c26-4881-bd77-798d533b7dd3\") " Oct 31 00:50:40.683210 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:40.683178 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e6cf90-2c26-4881-bd77-798d533b7dd3-maas-api-openshift-ca" pod "54e6cf90-2c26-4881-bd77-798d533b7dd3" (UID: "54e6cf90-2c26-4881-bd77-798d533b7dd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:40.684831 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:40.684808 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54e6cf90-2c26-4881-bd77-798d533b7dd3-maas-api-xks-ca" pod "54e6cf90-2c26-4881-bd77-798d533b7dd3" (UID: "54e6cf90-2c26-4881-bd77-798d533b7dd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:40.685015 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:40.684999 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e6cf90-2c26-4881-bd77-798d533b7dd3-kube-api-access-smnrb" pod "54e6cf90-2c26-4881-bd77-798d533b7dd3" (UID: "54e6cf90-2c26-4881-bd77-798d533b7dd3"). InnerVolumeSpecName "kube-api-access-smnrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:40.784329 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:40.784238 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smnrb\" (UniqueName: \"kubernetes.io/projected/54e6cf90-2c26-4881-bd77-798d533b7dd3-kube-api-access-smnrb\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:40.784329 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:40.784271 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e6cf90-2c26-4881-bd77-798d533b7dd3-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:40.784329 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:40.784280 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e6cf90-2c26-4881-bd77-798d533b7dd3-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:41.503479 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:41.503441 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" event={"ID":"54e6cf90-2c26-4881-bd77-798d533b7dd3","Type":"ContainerDied","Data":"30eea3b961eec86bf592212747125de96bce1cf21812076d87d06a07137e6917"} Oct 31 00:50:41.503479 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:41.503477 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw" Oct 31 00:50:41.503740 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:41.503528 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30eea3b961eec86bf592212747125de96bce1cf21812076d87d06a07137e6917" Oct 31 00:50:47.027014 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:47.026976 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7"] Oct 31 00:50:47.030507 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:47.030478 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e1d908f-hrkm7"] Oct 31 00:50:47.541971 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:50:47.541933 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:50:48.100904 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:48.100869 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41b7dd2d-ce74-4fb7-a39e-a0de7f78b978" path="/var/lib/kubelet/pods/41b7dd2d-ce74-4fb7-a39e-a0de7f78b978/volumes" Oct 31 00:50:55.888406 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:55.888370 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8"] Oct 31 00:50:55.888845 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:55.888743 4519 cpu_manager.go:414] "containerMap: removing container" podUID="54e6cf90-2c26-4881-bd77-798d533b7dd3" containerName="revoke-keys" Oct 31 00:50:55.888845 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:55.888756 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e6cf90-2c26-4881-bd77-798d533b7dd3" containerName="revoke-keys" Oct 31 00:50:55.888845 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:55.888838 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e6cf90-2c26-4881-bd77-798d533b7dd3" containerName="revoke-keys" Oct 31 00:50:55.891819 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:55.891804 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8" Oct 31 00:50:55.897865 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:55.897844 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:50:55.898200 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:55.898173 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:50:55.898369 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:55.898348 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:50:55.898499 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:55.898375 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:50:55.900409 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:55.900382 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8"] Oct 31 00:50:55.942325 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:55.942290 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56dce295-cd27-432a-aa4f-4e86ff4d070b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jmvm8\" (UID: \"56dce295-cd27-432a-aa4f-4e86ff4d070b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8" Oct 31 00:50:55.942521 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:55.942348 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xww2\" (UniqueName: \"kubernetes.io/projected/56dce295-cd27-432a-aa4f-4e86ff4d070b-kube-api-access-7xww2\") pod \"maas-api-revoke-sub-2e1f5b5f-jmvm8\" (UID: \"56dce295-cd27-432a-aa4f-4e86ff4d070b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8" Oct 31 00:50:55.942521 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:55.942430 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56dce295-cd27-432a-aa4f-4e86ff4d070b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jmvm8\" (UID: \"56dce295-cd27-432a-aa4f-4e86ff4d070b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8" Oct 31 00:50:56.043931 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:56.043885 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56dce295-cd27-432a-aa4f-4e86ff4d070b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jmvm8\" (UID: \"56dce295-cd27-432a-aa4f-4e86ff4d070b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8" Oct 31 00:50:56.044135 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:56.043959 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xww2\" (UniqueName: \"kubernetes.io/projected/56dce295-cd27-432a-aa4f-4e86ff4d070b-kube-api-access-7xww2\") pod \"maas-api-revoke-sub-2e1f5b5f-jmvm8\" (UID: \"56dce295-cd27-432a-aa4f-4e86ff4d070b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8" Oct 31 00:50:56.044135 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:56.043982 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56dce295-cd27-432a-aa4f-4e86ff4d070b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jmvm8\" (UID: \"56dce295-cd27-432a-aa4f-4e86ff4d070b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8" Oct 31 00:50:56.044540 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:56.044520 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56dce295-cd27-432a-aa4f-4e86ff4d070b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jmvm8\" (UID: \"56dce295-cd27-432a-aa4f-4e86ff4d070b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8" Oct 31 00:50:56.046363 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:56.046333 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56dce295-cd27-432a-aa4f-4e86ff4d070b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jmvm8\" (UID: \"56dce295-cd27-432a-aa4f-4e86ff4d070b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8" Oct 31 00:50:56.055267 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:56.055244 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xww2\" (UniqueName: \"kubernetes.io/projected/56dce295-cd27-432a-aa4f-4e86ff4d070b-kube-api-access-7xww2\") pod \"maas-api-revoke-sub-2e1f5b5f-jmvm8\" (UID: \"56dce295-cd27-432a-aa4f-4e86ff4d070b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8" Oct 31 00:50:56.207514 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:56.207430 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8" Oct 31 00:50:56.476537 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:56.476508 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8"] Oct 31 00:50:56.478620 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:50:56.478595 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56dce295_cd27_432a_aa4f_4e86ff4d070b.slice/crio-20479611331c1dac40cabb6ea377bc1c42e5a06f5e933834b61384408df66180 WatchSource:0}: Error finding container 20479611331c1dac40cabb6ea377bc1c42e5a06f5e933834b61384408df66180: Status 404 returned error can't find the container with id 20479611331c1dac40cabb6ea377bc1c42e5a06f5e933834b61384408df66180 Oct 31 00:50:56.581257 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:56.581224 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8" event={"ID":"56dce295-cd27-432a-aa4f-4e86ff4d070b","Type":"ContainerStarted","Data":"20479611331c1dac40cabb6ea377bc1c42e5a06f5e933834b61384408df66180"} Oct 31 00:50:57.587386 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:57.587284 4519 generic.go:356] "Generic (PLEG): container finished" podID="56dce295-cd27-432a-aa4f-4e86ff4d070b" containerID="b0c4fd71d53e386770d6094129160d9545aae598915b20ce0123578ea8356b83" exitCode=0 Oct 31 00:50:57.587386 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:57.587361 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8" event={"ID":"56dce295-cd27-432a-aa4f-4e86ff4d070b","Type":"ContainerDied","Data":"b0c4fd71d53e386770d6094129160d9545aae598915b20ce0123578ea8356b83"} Oct 31 00:50:58.026556 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:58.026514 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd"] Oct 31 00:50:58.029233 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:58.029204 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd9d6f7-qdnbd"] Oct 31 00:50:58.102081 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:58.102040 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1283fbf-f92e-465e-99f6-4582a8022574" path="/var/lib/kubelet/pods/a1283fbf-f92e-465e-99f6-4582a8022574/volumes" Oct 31 00:50:58.721367 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:58.721341 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8" Oct 31 00:50:58.769224 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:58.769187 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56dce295-cd27-432a-aa4f-4e86ff4d070b-kube-api-access-7xww2\" (UniqueName: \"kubernetes.io/projected/56dce295-cd27-432a-aa4f-4e86ff4d070b-kube-api-access-7xww2\") pod \"56dce295-cd27-432a-aa4f-4e86ff4d070b\" (UID: \"56dce295-cd27-432a-aa4f-4e86ff4d070b\") " Oct 31 00:50:58.769409 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:58.769269 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56dce295-cd27-432a-aa4f-4e86ff4d070b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56dce295-cd27-432a-aa4f-4e86ff4d070b-maas-api-xks-ca\") pod \"56dce295-cd27-432a-aa4f-4e86ff4d070b\" (UID: \"56dce295-cd27-432a-aa4f-4e86ff4d070b\") " Oct 31 00:50:58.769409 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:58.769293 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56dce295-cd27-432a-aa4f-4e86ff4d070b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56dce295-cd27-432a-aa4f-4e86ff4d070b-maas-api-openshift-ca\") pod \"56dce295-cd27-432a-aa4f-4e86ff4d070b\" (UID: \"56dce295-cd27-432a-aa4f-4e86ff4d070b\") " Oct 31 00:50:58.769702 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:58.769670 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56dce295-cd27-432a-aa4f-4e86ff4d070b-maas-api-openshift-ca" pod "56dce295-cd27-432a-aa4f-4e86ff4d070b" (UID: "56dce295-cd27-432a-aa4f-4e86ff4d070b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:58.771382 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:58.771358 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56dce295-cd27-432a-aa4f-4e86ff4d070b-maas-api-xks-ca" pod "56dce295-cd27-432a-aa4f-4e86ff4d070b" (UID: "56dce295-cd27-432a-aa4f-4e86ff4d070b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:58.771492 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:58.771456 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56dce295-cd27-432a-aa4f-4e86ff4d070b-kube-api-access-7xww2" pod "56dce295-cd27-432a-aa4f-4e86ff4d070b" (UID: "56dce295-cd27-432a-aa4f-4e86ff4d070b"). InnerVolumeSpecName "kube-api-access-7xww2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:58.870283 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:58.870192 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xww2\" (UniqueName: \"kubernetes.io/projected/56dce295-cd27-432a-aa4f-4e86ff4d070b-kube-api-access-7xww2\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:58.870283 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:58.870222 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56dce295-cd27-432a-aa4f-4e86ff4d070b-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:58.870283 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:58.870234 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56dce295-cd27-432a-aa4f-4e86ff4d070b-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:59.598106 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:59.598066 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8" event={"ID":"56dce295-cd27-432a-aa4f-4e86ff4d070b","Type":"ContainerDied","Data":"20479611331c1dac40cabb6ea377bc1c42e5a06f5e933834b61384408df66180"} Oct 31 00:50:59.598286 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:59.598115 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20479611331c1dac40cabb6ea377bc1c42e5a06f5e933834b61384408df66180" Oct 31 00:50:59.598286 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:50:59.598075 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8" Oct 31 00:51:06.820476 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:06.820437 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6"] Oct 31 00:51:06.820986 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:06.820766 4519 cpu_manager.go:414] "containerMap: removing container" podUID="56dce295-cd27-432a-aa4f-4e86ff4d070b" containerName="revoke-keys" Oct 31 00:51:06.820986 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:06.820777 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56dce295-cd27-432a-aa4f-4e86ff4d070b" containerName="revoke-keys" Oct 31 00:51:06.820986 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:06.820841 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="56dce295-cd27-432a-aa4f-4e86ff4d070b" containerName="revoke-keys" Oct 31 00:51:06.823973 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:06.823954 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6" Oct 31 00:51:06.831126 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:06.830863 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:51:06.831126 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:06.830869 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:51:06.831239 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:06.831155 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:51:06.831276 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:06.831247 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:51:06.832939 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:06.832915 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6"] Oct 31 00:51:06.946201 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:06.946162 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b5761b-cdcb-430f-bf12-899eb68a11d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4j9h6\" (UID: \"98b5761b-cdcb-430f-bf12-899eb68a11d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6" Oct 31 00:51:06.946390 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:06.946208 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb9wn\" (UniqueName: \"kubernetes.io/projected/98b5761b-cdcb-430f-bf12-899eb68a11d9-kube-api-access-hb9wn\") pod \"maas-api-revoke-sub-ec843018-4j9h6\" (UID: \"98b5761b-cdcb-430f-bf12-899eb68a11d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6" Oct 31 00:51:06.946390 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:06.946239 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b5761b-cdcb-430f-bf12-899eb68a11d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4j9h6\" (UID: \"98b5761b-cdcb-430f-bf12-899eb68a11d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6" Oct 31 00:51:07.047435 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:07.047390 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b5761b-cdcb-430f-bf12-899eb68a11d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4j9h6\" (UID: \"98b5761b-cdcb-430f-bf12-899eb68a11d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6" Oct 31 00:51:07.047608 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:07.047471 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9wn\" (UniqueName: \"kubernetes.io/projected/98b5761b-cdcb-430f-bf12-899eb68a11d9-kube-api-access-hb9wn\") pod \"maas-api-revoke-sub-ec843018-4j9h6\" (UID: \"98b5761b-cdcb-430f-bf12-899eb68a11d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6" Oct 31 00:51:07.047608 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:07.047523 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b5761b-cdcb-430f-bf12-899eb68a11d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4j9h6\" (UID: \"98b5761b-cdcb-430f-bf12-899eb68a11d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6" Oct 31 00:51:07.048239 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:07.048215 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b5761b-cdcb-430f-bf12-899eb68a11d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4j9h6\" (UID: \"98b5761b-cdcb-430f-bf12-899eb68a11d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6" Oct 31 00:51:07.049833 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:07.049812 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b5761b-cdcb-430f-bf12-899eb68a11d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4j9h6\" (UID: \"98b5761b-cdcb-430f-bf12-899eb68a11d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6" Oct 31 00:51:07.055626 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:07.055600 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb9wn\" (UniqueName: \"kubernetes.io/projected/98b5761b-cdcb-430f-bf12-899eb68a11d9-kube-api-access-hb9wn\") pod \"maas-api-revoke-sub-ec843018-4j9h6\" (UID: \"98b5761b-cdcb-430f-bf12-899eb68a11d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6" Oct 31 00:51:07.142129 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:07.142036 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6" Oct 31 00:51:07.429424 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:07.429398 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6"] Oct 31 00:51:07.431438 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:51:07.431406 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b5761b_cdcb_430f_bf12_899eb68a11d9.slice/crio-bcedebb5b5eb5a2a046782a2230cefa6523eb1b26c01bf1650465585acf22d8c WatchSource:0}: Error finding container bcedebb5b5eb5a2a046782a2230cefa6523eb1b26c01bf1650465585acf22d8c: Status 404 returned error can't find the container with id bcedebb5b5eb5a2a046782a2230cefa6523eb1b26c01bf1650465585acf22d8c Oct 31 00:51:07.638189 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:07.638160 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6" event={"ID":"98b5761b-cdcb-430f-bf12-899eb68a11d9","Type":"ContainerStarted","Data":"bcedebb5b5eb5a2a046782a2230cefa6523eb1b26c01bf1650465585acf22d8c"} Oct 31 00:51:08.644529 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:08.644439 4519 generic.go:356] "Generic (PLEG): container finished" podID="98b5761b-cdcb-430f-bf12-899eb68a11d9" containerID="c10e8f2a70c4130331beb4dd18c718100f87251bb68cff1b8da54fd233e89e56" exitCode=0 Oct 31 00:51:08.644529 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:08.644498 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6" event={"ID":"98b5761b-cdcb-430f-bf12-899eb68a11d9","Type":"ContainerDied","Data":"c10e8f2a70c4130331beb4dd18c718100f87251bb68cff1b8da54fd233e89e56"} Oct 31 00:51:10.010876 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:10.010852 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6" Oct 31 00:51:10.106967 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:10.106927 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98b5761b-cdcb-430f-bf12-899eb68a11d9-kube-api-access-hb9wn\" (UniqueName: \"kubernetes.io/projected/98b5761b-cdcb-430f-bf12-899eb68a11d9-kube-api-access-hb9wn\") pod \"98b5761b-cdcb-430f-bf12-899eb68a11d9\" (UID: \"98b5761b-cdcb-430f-bf12-899eb68a11d9\") " Oct 31 00:51:10.107167 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:10.107009 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98b5761b-cdcb-430f-bf12-899eb68a11d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b5761b-cdcb-430f-bf12-899eb68a11d9-maas-api-xks-ca\") pod \"98b5761b-cdcb-430f-bf12-899eb68a11d9\" (UID: \"98b5761b-cdcb-430f-bf12-899eb68a11d9\") " Oct 31 00:51:10.107167 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:10.107063 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98b5761b-cdcb-430f-bf12-899eb68a11d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b5761b-cdcb-430f-bf12-899eb68a11d9-maas-api-openshift-ca\") pod \"98b5761b-cdcb-430f-bf12-899eb68a11d9\" (UID: \"98b5761b-cdcb-430f-bf12-899eb68a11d9\") " Oct 31 00:51:10.107502 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:10.107468 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b5761b-cdcb-430f-bf12-899eb68a11d9-maas-api-openshift-ca" pod "98b5761b-cdcb-430f-bf12-899eb68a11d9" (UID: "98b5761b-cdcb-430f-bf12-899eb68a11d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:10.109687 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:10.109652 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b5761b-cdcb-430f-bf12-899eb68a11d9-maas-api-xks-ca" pod "98b5761b-cdcb-430f-bf12-899eb68a11d9" (UID: "98b5761b-cdcb-430f-bf12-899eb68a11d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:10.109965 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:10.109946 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b5761b-cdcb-430f-bf12-899eb68a11d9-kube-api-access-hb9wn" pod "98b5761b-cdcb-430f-bf12-899eb68a11d9" (UID: "98b5761b-cdcb-430f-bf12-899eb68a11d9"). InnerVolumeSpecName "kube-api-access-hb9wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:10.207910 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:10.207871 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb9wn\" (UniqueName: \"kubernetes.io/projected/98b5761b-cdcb-430f-bf12-899eb68a11d9-kube-api-access-hb9wn\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:10.207910 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:10.207904 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b5761b-cdcb-430f-bf12-899eb68a11d9-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:10.207910 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:10.207919 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b5761b-cdcb-430f-bf12-899eb68a11d9-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:10.697881 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:10.697841 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6" event={"ID":"98b5761b-cdcb-430f-bf12-899eb68a11d9","Type":"ContainerDied","Data":"bcedebb5b5eb5a2a046782a2230cefa6523eb1b26c01bf1650465585acf22d8c"} Oct 31 00:51:10.698092 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:10.697908 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcedebb5b5eb5a2a046782a2230cefa6523eb1b26c01bf1650465585acf22d8c" Oct 31 00:51:10.698092 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:10.697977 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6" Oct 31 00:51:10.801887 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:10.801839 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8wb6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:51:10.802067 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:10.801905 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" podUID="a11bc6fe-fa8b-425e-b6df-007d3fc6aa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:51:11.663439 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:11.663408 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:17.540715 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:51:17.540677 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:51:25.955742 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:25.955705 4519 scope.go:122] "RemoveContainer" containerID="9ae13c5926dd2434c170467123cb87054517b69d82dcbae50b7c3bcd97e207a1" Oct 31 00:51:25.964251 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:25.964229 4519 scope.go:122] "RemoveContainer" containerID="f4bbad515880662dfeda4693cf930dc7cc85e86dd86e1b5cf47d161202644e75" Oct 31 00:51:25.972091 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:25.972067 4519 scope.go:122] "RemoveContainer" containerID="0daf1901252d23296a1526169e67a9eb2a0d0eb1c935d0d1bc7dc81bd9d53198" Oct 31 00:51:33.047155 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.047119 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks"] Oct 31 00:51:33.047555 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.047455 4519 cpu_manager.go:414] "containerMap: removing container" podUID="98b5761b-cdcb-430f-bf12-899eb68a11d9" containerName="revoke-keys" Oct 31 00:51:33.047555 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.047466 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98b5761b-cdcb-430f-bf12-899eb68a11d9" containerName="revoke-keys" Oct 31 00:51:33.047555 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.047519 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="98b5761b-cdcb-430f-bf12-899eb68a11d9" containerName="revoke-keys" Oct 31 00:51:33.052034 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.052017 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks" Oct 31 00:51:33.056761 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.056732 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:51:33.058155 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.058122 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:51:33.058302 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.058285 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:51:33.058382 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.058335 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:51:33.059576 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.059396 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks"] Oct 31 00:51:33.118627 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.118585 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sczq6\" (UniqueName: \"kubernetes.io/projected/273cc646-5b6e-4ab0-892d-c37d35ab4995-kube-api-access-sczq6\") pod \"maas-api-revoke-sub-046e1343-9bmks\" (UID: \"273cc646-5b6e-4ab0-892d-c37d35ab4995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks" Oct 31 00:51:33.118849 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.118733 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273cc646-5b6e-4ab0-892d-c37d35ab4995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-9bmks\" (UID: \"273cc646-5b6e-4ab0-892d-c37d35ab4995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks" Oct 31 00:51:33.118849 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.118788 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273cc646-5b6e-4ab0-892d-c37d35ab4995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-9bmks\" (UID: \"273cc646-5b6e-4ab0-892d-c37d35ab4995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks" Oct 31 00:51:33.219701 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.219662 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczq6\" (UniqueName: \"kubernetes.io/projected/273cc646-5b6e-4ab0-892d-c37d35ab4995-kube-api-access-sczq6\") pod \"maas-api-revoke-sub-046e1343-9bmks\" (UID: \"273cc646-5b6e-4ab0-892d-c37d35ab4995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks" Oct 31 00:51:33.219889 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.219732 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273cc646-5b6e-4ab0-892d-c37d35ab4995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-9bmks\" (UID: \"273cc646-5b6e-4ab0-892d-c37d35ab4995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks" Oct 31 00:51:33.219889 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.219758 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273cc646-5b6e-4ab0-892d-c37d35ab4995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-9bmks\" (UID: \"273cc646-5b6e-4ab0-892d-c37d35ab4995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks" Oct 31 00:51:33.220449 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.220419 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273cc646-5b6e-4ab0-892d-c37d35ab4995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-9bmks\" (UID: \"273cc646-5b6e-4ab0-892d-c37d35ab4995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks" Oct 31 00:51:33.222254 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.222231 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273cc646-5b6e-4ab0-892d-c37d35ab4995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-9bmks\" (UID: \"273cc646-5b6e-4ab0-892d-c37d35ab4995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks" Oct 31 00:51:33.233220 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.233193 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sczq6\" (UniqueName: \"kubernetes.io/projected/273cc646-5b6e-4ab0-892d-c37d35ab4995-kube-api-access-sczq6\") pod \"maas-api-revoke-sub-046e1343-9bmks\" (UID: \"273cc646-5b6e-4ab0-892d-c37d35ab4995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks" Oct 31 00:51:33.366397 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.366295 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks" Oct 31 00:51:33.637141 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.637112 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks"] Oct 31 00:51:33.639055 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:51:33.639030 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273cc646_5b6e_4ab0_892d_c37d35ab4995.slice/crio-dedd9d356d498d0bec357bf045044298f99650280f7b60acd1b7a47fb46fa03e WatchSource:0}: Error finding container dedd9d356d498d0bec357bf045044298f99650280f7b60acd1b7a47fb46fa03e: Status 404 returned error can't find the container with id dedd9d356d498d0bec357bf045044298f99650280f7b60acd1b7a47fb46fa03e Oct 31 00:51:33.842487 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:33.842457 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks" event={"ID":"273cc646-5b6e-4ab0-892d-c37d35ab4995","Type":"ContainerStarted","Data":"dedd9d356d498d0bec357bf045044298f99650280f7b60acd1b7a47fb46fa03e"} Oct 31 00:51:34.853009 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:34.852972 4519 generic.go:356] "Generic (PLEG): container finished" podID="273cc646-5b6e-4ab0-892d-c37d35ab4995" containerID="539644a855408e2bab4c10fc1398c1ff11ea1d8e8c65b82c4cd4e0ccfb62f3d4" exitCode=0 Oct 31 00:51:34.853419 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:34.853060 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks" event={"ID":"273cc646-5b6e-4ab0-892d-c37d35ab4995","Type":"ContainerDied","Data":"539644a855408e2bab4c10fc1398c1ff11ea1d8e8c65b82c4cd4e0ccfb62f3d4"} Oct 31 00:51:35.988247 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:35.988219 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks" Oct 31 00:51:36.046935 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:36.046893 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/273cc646-5b6e-4ab0-892d-c37d35ab4995-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273cc646-5b6e-4ab0-892d-c37d35ab4995-maas-api-xks-ca\") pod \"273cc646-5b6e-4ab0-892d-c37d35ab4995\" (UID: \"273cc646-5b6e-4ab0-892d-c37d35ab4995\") " Oct 31 00:51:36.047122 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:36.046953 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/273cc646-5b6e-4ab0-892d-c37d35ab4995-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273cc646-5b6e-4ab0-892d-c37d35ab4995-maas-api-openshift-ca\") pod \"273cc646-5b6e-4ab0-892d-c37d35ab4995\" (UID: \"273cc646-5b6e-4ab0-892d-c37d35ab4995\") " Oct 31 00:51:36.047122 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:36.046987 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273cc646-5b6e-4ab0-892d-c37d35ab4995-kube-api-access-sczq6\" (UniqueName: \"kubernetes.io/projected/273cc646-5b6e-4ab0-892d-c37d35ab4995-kube-api-access-sczq6\") pod \"273cc646-5b6e-4ab0-892d-c37d35ab4995\" (UID: \"273cc646-5b6e-4ab0-892d-c37d35ab4995\") " Oct 31 00:51:36.047366 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:36.047337 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273cc646-5b6e-4ab0-892d-c37d35ab4995-maas-api-openshift-ca" pod "273cc646-5b6e-4ab0-892d-c37d35ab4995" (UID: "273cc646-5b6e-4ab0-892d-c37d35ab4995"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:36.049085 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:36.049061 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/273cc646-5b6e-4ab0-892d-c37d35ab4995-maas-api-xks-ca" pod "273cc646-5b6e-4ab0-892d-c37d35ab4995" (UID: "273cc646-5b6e-4ab0-892d-c37d35ab4995"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:36.049171 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:36.049139 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273cc646-5b6e-4ab0-892d-c37d35ab4995-kube-api-access-sczq6" pod "273cc646-5b6e-4ab0-892d-c37d35ab4995" (UID: "273cc646-5b6e-4ab0-892d-c37d35ab4995"). InnerVolumeSpecName "kube-api-access-sczq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:36.147956 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:36.147874 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273cc646-5b6e-4ab0-892d-c37d35ab4995-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:36.147956 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:36.147908 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273cc646-5b6e-4ab0-892d-c37d35ab4995-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:36.147956 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:36.147920 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sczq6\" (UniqueName: \"kubernetes.io/projected/273cc646-5b6e-4ab0-892d-c37d35ab4995-kube-api-access-sczq6\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:36.864950 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:36.864918 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks" Oct 31 00:51:36.865109 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:36.864918 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks" event={"ID":"273cc646-5b6e-4ab0-892d-c37d35ab4995","Type":"ContainerDied","Data":"dedd9d356d498d0bec357bf045044298f99650280f7b60acd1b7a47fb46fa03e"} Oct 31 00:51:36.865109 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:36.865036 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dedd9d356d498d0bec357bf045044298f99650280f7b60acd1b7a47fb46fa03e" Oct 31 00:51:43.244703 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.244668 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr"] Oct 31 00:51:43.245356 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.244992 4519 cpu_manager.go:414] "containerMap: removing container" podUID="273cc646-5b6e-4ab0-892d-c37d35ab4995" containerName="revoke-keys" Oct 31 00:51:43.245356 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.245003 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273cc646-5b6e-4ab0-892d-c37d35ab4995" containerName="revoke-keys" Oct 31 00:51:43.245356 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.245062 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="273cc646-5b6e-4ab0-892d-c37d35ab4995" containerName="revoke-keys" Oct 31 00:51:43.248008 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.247991 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr" Oct 31 00:51:43.254909 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.254737 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:51:43.256053 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.256027 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:51:43.256217 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.256034 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:51:43.256217 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.256102 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:51:43.258738 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.258714 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr"] Oct 31 00:51:43.308234 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.308198 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4mbx\" (UniqueName: \"kubernetes.io/projected/51b515b8-c37e-4630-abe4-0cbdcbdbc084-kube-api-access-r4mbx\") pod \"maas-api-revoke-sub-d3fba056-x9rjr\" (UID: \"51b515b8-c37e-4630-abe4-0cbdcbdbc084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr" Oct 31 00:51:43.308444 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.308247 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b515b8-c37e-4630-abe4-0cbdcbdbc084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-x9rjr\" (UID: \"51b515b8-c37e-4630-abe4-0cbdcbdbc084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr" Oct 31 00:51:43.308444 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.308266 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b515b8-c37e-4630-abe4-0cbdcbdbc084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-x9rjr\" (UID: \"51b515b8-c37e-4630-abe4-0cbdcbdbc084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr" Oct 31 00:51:43.409102 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.409064 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b515b8-c37e-4630-abe4-0cbdcbdbc084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-x9rjr\" (UID: \"51b515b8-c37e-4630-abe4-0cbdcbdbc084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr" Oct 31 00:51:43.409102 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.409102 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b515b8-c37e-4630-abe4-0cbdcbdbc084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-x9rjr\" (UID: \"51b515b8-c37e-4630-abe4-0cbdcbdbc084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr" Oct 31 00:51:43.409325 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.409178 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4mbx\" (UniqueName: \"kubernetes.io/projected/51b515b8-c37e-4630-abe4-0cbdcbdbc084-kube-api-access-r4mbx\") pod \"maas-api-revoke-sub-d3fba056-x9rjr\" (UID: \"51b515b8-c37e-4630-abe4-0cbdcbdbc084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr" Oct 31 00:51:43.409760 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.409731 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b515b8-c37e-4630-abe4-0cbdcbdbc084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-x9rjr\" (UID: \"51b515b8-c37e-4630-abe4-0cbdcbdbc084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr" Oct 31 00:51:43.411520 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.411501 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b515b8-c37e-4630-abe4-0cbdcbdbc084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-x9rjr\" (UID: \"51b515b8-c37e-4630-abe4-0cbdcbdbc084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr" Oct 31 00:51:43.417724 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.417697 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4mbx\" (UniqueName: \"kubernetes.io/projected/51b515b8-c37e-4630-abe4-0cbdcbdbc084-kube-api-access-r4mbx\") pod \"maas-api-revoke-sub-d3fba056-x9rjr\" (UID: \"51b515b8-c37e-4630-abe4-0cbdcbdbc084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr" Oct 31 00:51:43.561343 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.561258 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr" Oct 31 00:51:43.832364 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.832333 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr"] Oct 31 00:51:43.834566 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:51:43.834521 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b515b8_c37e_4630_abe4_0cbdcbdbc084.slice/crio-058da624650116c0840e36459f8fbe56ccbb7fb5577e27454aa34c9c14373776 WatchSource:0}: Error finding container 058da624650116c0840e36459f8fbe56ccbb7fb5577e27454aa34c9c14373776: Status 404 returned error can't find the container with id 058da624650116c0840e36459f8fbe56ccbb7fb5577e27454aa34c9c14373776 Oct 31 00:51:43.900236 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:43.900201 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr" event={"ID":"51b515b8-c37e-4630-abe4-0cbdcbdbc084","Type":"ContainerStarted","Data":"058da624650116c0840e36459f8fbe56ccbb7fb5577e27454aa34c9c14373776"} Oct 31 00:51:44.906867 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:44.906816 4519 generic.go:356] "Generic (PLEG): container finished" podID="51b515b8-c37e-4630-abe4-0cbdcbdbc084" containerID="4e679e4922d0498f2b93ec8f9b253248dffcbec653a0119a005e0471f5920119" exitCode=0 Oct 31 00:51:44.907257 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:44.906884 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr" event={"ID":"51b515b8-c37e-4630-abe4-0cbdcbdbc084","Type":"ContainerDied","Data":"4e679e4922d0498f2b93ec8f9b253248dffcbec653a0119a005e0471f5920119"} Oct 31 00:51:46.222157 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:46.222129 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr" Oct 31 00:51:46.403063 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:46.403025 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b515b8-c37e-4630-abe4-0cbdcbdbc084-kube-api-access-r4mbx\" (UniqueName: \"kubernetes.io/projected/51b515b8-c37e-4630-abe4-0cbdcbdbc084-kube-api-access-r4mbx\") pod \"51b515b8-c37e-4630-abe4-0cbdcbdbc084\" (UID: \"51b515b8-c37e-4630-abe4-0cbdcbdbc084\") " Oct 31 00:51:46.403063 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:46.403070 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51b515b8-c37e-4630-abe4-0cbdcbdbc084-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b515b8-c37e-4630-abe4-0cbdcbdbc084-maas-api-xks-ca\") pod \"51b515b8-c37e-4630-abe4-0cbdcbdbc084\" (UID: \"51b515b8-c37e-4630-abe4-0cbdcbdbc084\") " Oct 31 00:51:46.403341 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:46.403115 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51b515b8-c37e-4630-abe4-0cbdcbdbc084-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b515b8-c37e-4630-abe4-0cbdcbdbc084-maas-api-openshift-ca\") pod \"51b515b8-c37e-4630-abe4-0cbdcbdbc084\" (UID: \"51b515b8-c37e-4630-abe4-0cbdcbdbc084\") " Oct 31 00:51:46.403538 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:46.403507 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b515b8-c37e-4630-abe4-0cbdcbdbc084-maas-api-openshift-ca" pod "51b515b8-c37e-4630-abe4-0cbdcbdbc084" (UID: "51b515b8-c37e-4630-abe4-0cbdcbdbc084"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:46.405255 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:46.405221 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b515b8-c37e-4630-abe4-0cbdcbdbc084-maas-api-xks-ca" pod "51b515b8-c37e-4630-abe4-0cbdcbdbc084" (UID: "51b515b8-c37e-4630-abe4-0cbdcbdbc084"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:46.405356 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:46.405231 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b515b8-c37e-4630-abe4-0cbdcbdbc084-kube-api-access-r4mbx" pod "51b515b8-c37e-4630-abe4-0cbdcbdbc084" (UID: "51b515b8-c37e-4630-abe4-0cbdcbdbc084"). InnerVolumeSpecName "kube-api-access-r4mbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:46.504341 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:46.504300 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4mbx\" (UniqueName: \"kubernetes.io/projected/51b515b8-c37e-4630-abe4-0cbdcbdbc084-kube-api-access-r4mbx\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:46.504341 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:46.504335 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b515b8-c37e-4630-abe4-0cbdcbdbc084-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:46.504341 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:46.504346 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b515b8-c37e-4630-abe4-0cbdcbdbc084-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:46.918087 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:46.917996 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr" Oct 31 00:51:46.918087 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:46.918038 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr" event={"ID":"51b515b8-c37e-4630-abe4-0cbdcbdbc084","Type":"ContainerDied","Data":"058da624650116c0840e36459f8fbe56ccbb7fb5577e27454aa34c9c14373776"} Oct 31 00:51:46.918347 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:51:46.918098 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="058da624650116c0840e36459f8fbe56ccbb7fb5577e27454aa34c9c14373776" Oct 31 00:51:47.541653 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:51:47.541607 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:52:04.026487 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.026454 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq"] Oct 31 00:52:04.026926 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.026779 4519 cpu_manager.go:414] "containerMap: removing container" podUID="51b515b8-c37e-4630-abe4-0cbdcbdbc084" containerName="revoke-keys" Oct 31 00:52:04.026926 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.026791 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b515b8-c37e-4630-abe4-0cbdcbdbc084" containerName="revoke-keys" Oct 31 00:52:04.026926 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.026863 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b515b8-c37e-4630-abe4-0cbdcbdbc084" containerName="revoke-keys" Oct 31 00:52:04.029863 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.029847 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" Oct 31 00:52:04.035242 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.035093 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:52:04.036742 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.036709 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:52:04.036872 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.036796 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:52:04.036945 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.036889 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:52:04.039331 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.039308 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq"] Oct 31 00:52:04.056262 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.056225 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzbjt\" (UniqueName: \"kubernetes.io/projected/8cf6b86a-2431-4443-90fe-56e2956b230e-kube-api-access-bzbjt\") pod \"maas-api-revoke-sub-ae594979-nr7bq\" (UID: \"8cf6b86a-2431-4443-90fe-56e2956b230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" Oct 31 00:52:04.056262 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.056264 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf6b86a-2431-4443-90fe-56e2956b230e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nr7bq\" (UID: \"8cf6b86a-2431-4443-90fe-56e2956b230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" Oct 31 00:52:04.056480 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.056343 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf6b86a-2431-4443-90fe-56e2956b230e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nr7bq\" (UID: \"8cf6b86a-2431-4443-90fe-56e2956b230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" Oct 31 00:52:04.157170 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.157136 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzbjt\" (UniqueName: \"kubernetes.io/projected/8cf6b86a-2431-4443-90fe-56e2956b230e-kube-api-access-bzbjt\") pod \"maas-api-revoke-sub-ae594979-nr7bq\" (UID: \"8cf6b86a-2431-4443-90fe-56e2956b230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" Oct 31 00:52:04.157344 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.157179 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf6b86a-2431-4443-90fe-56e2956b230e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nr7bq\" (UID: \"8cf6b86a-2431-4443-90fe-56e2956b230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" Oct 31 00:52:04.157344 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.157223 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf6b86a-2431-4443-90fe-56e2956b230e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nr7bq\" (UID: \"8cf6b86a-2431-4443-90fe-56e2956b230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" Oct 31 00:52:04.157905 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.157880 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf6b86a-2431-4443-90fe-56e2956b230e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nr7bq\" (UID: \"8cf6b86a-2431-4443-90fe-56e2956b230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" Oct 31 00:52:04.159539 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.159518 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf6b86a-2431-4443-90fe-56e2956b230e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nr7bq\" (UID: \"8cf6b86a-2431-4443-90fe-56e2956b230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" Oct 31 00:52:04.166763 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.166740 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzbjt\" (UniqueName: \"kubernetes.io/projected/8cf6b86a-2431-4443-90fe-56e2956b230e-kube-api-access-bzbjt\") pod \"maas-api-revoke-sub-ae594979-nr7bq\" (UID: \"8cf6b86a-2431-4443-90fe-56e2956b230e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" Oct 31 00:52:04.345266 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.345175 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" Oct 31 00:52:04.616766 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:04.616741 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq"] Oct 31 00:52:04.618777 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:52:04.618747 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf6b86a_2431_4443_90fe_56e2956b230e.slice/crio-aaaae61787272030199aa780d9e27a7cc600628ecc8448688b1b4ed4d79befbb WatchSource:0}: Error finding container aaaae61787272030199aa780d9e27a7cc600628ecc8448688b1b4ed4d79befbb: Status 404 returned error can't find the container with id aaaae61787272030199aa780d9e27a7cc600628ecc8448688b1b4ed4d79befbb Oct 31 00:52:05.028701 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:05.028657 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" event={"ID":"8cf6b86a-2431-4443-90fe-56e2956b230e","Type":"ContainerStarted","Data":"de76857a481490370a11fc42c69a654b73e75f9ea81e0482b318166d7f58def9"} Oct 31 00:52:05.029079 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:05.028713 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" event={"ID":"8cf6b86a-2431-4443-90fe-56e2956b230e","Type":"ContainerStarted","Data":"aaaae61787272030199aa780d9e27a7cc600628ecc8448688b1b4ed4d79befbb"} Oct 31 00:52:05.047145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:05.047091 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" podStartSLOduration=1.04707805 podStartE2EDuration="1.04707805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:05.046168552 +0000 UTC m=+1121.075348942" watchObservedRunningTime="2026-10-31 00:52:05.04707805 +0000 UTC m=+1121.076258439" Oct 31 00:52:06.034756 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:06.034678 4519 generic.go:356] "Generic (PLEG): container finished" podID="8cf6b86a-2431-4443-90fe-56e2956b230e" containerID="de76857a481490370a11fc42c69a654b73e75f9ea81e0482b318166d7f58def9" exitCode=0 Oct 31 00:52:06.035128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:06.034759 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" event={"ID":"8cf6b86a-2431-4443-90fe-56e2956b230e","Type":"ContainerDied","Data":"de76857a481490370a11fc42c69a654b73e75f9ea81e0482b318166d7f58def9"} Oct 31 00:52:07.160775 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:07.160750 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" Oct 31 00:52:07.184248 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:07.184210 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf6b86a-2431-4443-90fe-56e2956b230e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf6b86a-2431-4443-90fe-56e2956b230e-maas-api-openshift-ca\") pod \"8cf6b86a-2431-4443-90fe-56e2956b230e\" (UID: \"8cf6b86a-2431-4443-90fe-56e2956b230e\") " Oct 31 00:52:07.184558 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:07.184539 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf6b86a-2431-4443-90fe-56e2956b230e-kube-api-access-bzbjt\" (UniqueName: \"kubernetes.io/projected/8cf6b86a-2431-4443-90fe-56e2956b230e-kube-api-access-bzbjt\") pod \"8cf6b86a-2431-4443-90fe-56e2956b230e\" (UID: \"8cf6b86a-2431-4443-90fe-56e2956b230e\") " Oct 31 00:52:07.184740 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:07.184722 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cf6b86a-2431-4443-90fe-56e2956b230e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf6b86a-2431-4443-90fe-56e2956b230e-maas-api-xks-ca\") pod \"8cf6b86a-2431-4443-90fe-56e2956b230e\" (UID: \"8cf6b86a-2431-4443-90fe-56e2956b230e\") " Oct 31 00:52:07.185961 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:07.185931 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf6b86a-2431-4443-90fe-56e2956b230e-maas-api-openshift-ca" pod "8cf6b86a-2431-4443-90fe-56e2956b230e" (UID: "8cf6b86a-2431-4443-90fe-56e2956b230e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:07.189937 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:07.189910 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf6b86a-2431-4443-90fe-56e2956b230e-kube-api-access-bzbjt" pod "8cf6b86a-2431-4443-90fe-56e2956b230e" (UID: "8cf6b86a-2431-4443-90fe-56e2956b230e"). InnerVolumeSpecName "kube-api-access-bzbjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:07.190053 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:07.189930 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf6b86a-2431-4443-90fe-56e2956b230e-maas-api-xks-ca" pod "8cf6b86a-2431-4443-90fe-56e2956b230e" (UID: "8cf6b86a-2431-4443-90fe-56e2956b230e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:52:07.286563 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:07.286515 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf6b86a-2431-4443-90fe-56e2956b230e-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:07.286563 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:07.286559 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf6b86a-2431-4443-90fe-56e2956b230e-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:07.286563 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:07.286570 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzbjt\" (UniqueName: \"kubernetes.io/projected/8cf6b86a-2431-4443-90fe-56e2956b230e-kube-api-access-bzbjt\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:08.045605 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:08.045569 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" Oct 31 00:52:08.045800 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:08.045604 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq" event={"ID":"8cf6b86a-2431-4443-90fe-56e2956b230e","Type":"ContainerDied","Data":"aaaae61787272030199aa780d9e27a7cc600628ecc8448688b1b4ed4d79befbb"} Oct 31 00:52:08.045800 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:08.045702 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaaae61787272030199aa780d9e27a7cc600628ecc8448688b1b4ed4d79befbb" Oct 31 00:52:11.720223 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:11.720157 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:17.544924 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:52:17.544877 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:52:22.402709 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.402671 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47"] Oct 31 00:52:22.403971 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.403943 4519 cpu_manager.go:414] "containerMap: removing container" podUID="8cf6b86a-2431-4443-90fe-56e2956b230e" containerName="revoke-keys" Oct 31 00:52:22.404131 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.404118 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf6b86a-2431-4443-90fe-56e2956b230e" containerName="revoke-keys" Oct 31 00:52:22.404353 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.404341 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf6b86a-2431-4443-90fe-56e2956b230e" containerName="revoke-keys" Oct 31 00:52:22.408221 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.408199 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47" Oct 31 00:52:22.412261 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.412230 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47"] Oct 31 00:52:22.412464 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.412447 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:52:22.412545 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.412513 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:52:22.413518 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.413501 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:52:22.413613 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.413546 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:52:22.538261 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.538226 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b6f96e-3e32-45a0-8648-647b2f58ab38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zrw47\" (UID: \"f1b6f96e-3e32-45a0-8648-647b2f58ab38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47" Oct 31 00:52:22.538429 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.538268 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b6f96e-3e32-45a0-8648-647b2f58ab38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zrw47\" (UID: \"f1b6f96e-3e32-45a0-8648-647b2f58ab38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47" Oct 31 00:52:22.538429 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.538345 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts8rh\" (UniqueName: \"kubernetes.io/projected/f1b6f96e-3e32-45a0-8648-647b2f58ab38-kube-api-access-ts8rh\") pod \"maas-api-revoke-sub-a4aef7d7-zrw47\" (UID: \"f1b6f96e-3e32-45a0-8648-647b2f58ab38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47" Oct 31 00:52:22.639284 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.639257 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b6f96e-3e32-45a0-8648-647b2f58ab38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zrw47\" (UID: \"f1b6f96e-3e32-45a0-8648-647b2f58ab38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47" Oct 31 00:52:22.639470 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.639297 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b6f96e-3e32-45a0-8648-647b2f58ab38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zrw47\" (UID: \"f1b6f96e-3e32-45a0-8648-647b2f58ab38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47" Oct 31 00:52:22.639470 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.639410 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts8rh\" (UniqueName: \"kubernetes.io/projected/f1b6f96e-3e32-45a0-8648-647b2f58ab38-kube-api-access-ts8rh\") pod \"maas-api-revoke-sub-a4aef7d7-zrw47\" (UID: \"f1b6f96e-3e32-45a0-8648-647b2f58ab38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47" Oct 31 00:52:22.640068 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.640041 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b6f96e-3e32-45a0-8648-647b2f58ab38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zrw47\" (UID: \"f1b6f96e-3e32-45a0-8648-647b2f58ab38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47" Oct 31 00:52:22.641426 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.641405 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b6f96e-3e32-45a0-8648-647b2f58ab38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zrw47\" (UID: \"f1b6f96e-3e32-45a0-8648-647b2f58ab38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47" Oct 31 00:52:22.648423 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.648402 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts8rh\" (UniqueName: \"kubernetes.io/projected/f1b6f96e-3e32-45a0-8648-647b2f58ab38-kube-api-access-ts8rh\") pod \"maas-api-revoke-sub-a4aef7d7-zrw47\" (UID: \"f1b6f96e-3e32-45a0-8648-647b2f58ab38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47" Oct 31 00:52:22.723443 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.723351 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47" Oct 31 00:52:22.993075 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:22.993047 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47"] Oct 31 00:52:22.995180 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:52:22.995141 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b6f96e_3e32_45a0_8648_647b2f58ab38.slice/crio-03bc9333552daee9db6d0c316f22e1b229ff62dbcc9a13f012e3234f6e800125 WatchSource:0}: Error finding container 03bc9333552daee9db6d0c316f22e1b229ff62dbcc9a13f012e3234f6e800125: Status 404 returned error can't find the container with id 03bc9333552daee9db6d0c316f22e1b229ff62dbcc9a13f012e3234f6e800125 Oct 31 00:52:23.121988 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:23.121953 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47" event={"ID":"f1b6f96e-3e32-45a0-8648-647b2f58ab38","Type":"ContainerStarted","Data":"03bc9333552daee9db6d0c316f22e1b229ff62dbcc9a13f012e3234f6e800125"} Oct 31 00:52:24.127889 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:24.127811 4519 generic.go:356] "Generic (PLEG): container finished" podID="f1b6f96e-3e32-45a0-8648-647b2f58ab38" containerID="7ff15474d8056e1dd8158af12619b0c98c953bfb3e7ae9ce5ae6ebfc53fe6733" exitCode=0 Oct 31 00:52:24.128245 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:24.127895 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47" event={"ID":"f1b6f96e-3e32-45a0-8648-647b2f58ab38","Type":"ContainerDied","Data":"7ff15474d8056e1dd8158af12619b0c98c953bfb3e7ae9ce5ae6ebfc53fe6733"} Oct 31 00:52:25.254188 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:25.254164 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47" Oct 31 00:52:25.364875 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:25.364832 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1b6f96e-3e32-45a0-8648-647b2f58ab38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b6f96e-3e32-45a0-8648-647b2f58ab38-maas-api-xks-ca\") pod \"f1b6f96e-3e32-45a0-8648-647b2f58ab38\" (UID: \"f1b6f96e-3e32-45a0-8648-647b2f58ab38\") " Oct 31 00:52:25.365080 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:25.364898 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1b6f96e-3e32-45a0-8648-647b2f58ab38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b6f96e-3e32-45a0-8648-647b2f58ab38-maas-api-openshift-ca\") pod \"f1b6f96e-3e32-45a0-8648-647b2f58ab38\" (UID: \"f1b6f96e-3e32-45a0-8648-647b2f58ab38\") " Oct 31 00:52:25.365080 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:25.364989 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b6f96e-3e32-45a0-8648-647b2f58ab38-kube-api-access-ts8rh\" (UniqueName: \"kubernetes.io/projected/f1b6f96e-3e32-45a0-8648-647b2f58ab38-kube-api-access-ts8rh\") pod \"f1b6f96e-3e32-45a0-8648-647b2f58ab38\" (UID: \"f1b6f96e-3e32-45a0-8648-647b2f58ab38\") " Oct 31 00:52:25.365260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:25.365230 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b6f96e-3e32-45a0-8648-647b2f58ab38-maas-api-openshift-ca" pod "f1b6f96e-3e32-45a0-8648-647b2f58ab38" (UID: "f1b6f96e-3e32-45a0-8648-647b2f58ab38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:25.367089 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:25.367058 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b6f96e-3e32-45a0-8648-647b2f58ab38-kube-api-access-ts8rh" pod "f1b6f96e-3e32-45a0-8648-647b2f58ab38" (UID: "f1b6f96e-3e32-45a0-8648-647b2f58ab38"). InnerVolumeSpecName "kube-api-access-ts8rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:25.367199 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:25.367107 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b6f96e-3e32-45a0-8648-647b2f58ab38-maas-api-xks-ca" pod "f1b6f96e-3e32-45a0-8648-647b2f58ab38" (UID: "f1b6f96e-3e32-45a0-8648-647b2f58ab38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:52:25.466480 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:25.466390 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts8rh\" (UniqueName: \"kubernetes.io/projected/f1b6f96e-3e32-45a0-8648-647b2f58ab38-kube-api-access-ts8rh\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:25.466480 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:25.466425 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b6f96e-3e32-45a0-8648-647b2f58ab38-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:25.466480 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:25.466438 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b6f96e-3e32-45a0-8648-647b2f58ab38-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:25.999944 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:25.999908 4519 scope.go:122] "RemoveContainer" containerID="e2f6ceae620ace523f8d69c9f1e3378818809415378c8f76d87a0135f23ab46a" Oct 31 00:52:26.014366 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:26.014340 4519 scope.go:122] "RemoveContainer" containerID="fa0c0f5a7fc362a695939a6fda65ce4f63a18d3d65090afed4a061e3372d4f70" Oct 31 00:52:26.022489 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:26.022467 4519 scope.go:122] "RemoveContainer" containerID="11d440a8f9eaa393d80dd02fc33acc8d198d4b765515323b7823f437ef09e30a" Oct 31 00:52:26.030203 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:26.030182 4519 scope.go:122] "RemoveContainer" containerID="b813adb67d791046ec39734c8e109578e131b3d900a29482f238258644d7d56e" Oct 31 00:52:26.138860 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:26.138817 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47" event={"ID":"f1b6f96e-3e32-45a0-8648-647b2f58ab38","Type":"ContainerDied","Data":"03bc9333552daee9db6d0c316f22e1b229ff62dbcc9a13f012e3234f6e800125"} Oct 31 00:52:26.138860 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:26.138853 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47" Oct 31 00:52:26.139070 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:26.138879 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03bc9333552daee9db6d0c316f22e1b229ff62dbcc9a13f012e3234f6e800125" Oct 31 00:52:41.160025 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.159985 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh"] Oct 31 00:52:41.160472 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.160339 4519 cpu_manager.go:414] "containerMap: removing container" podUID="f1b6f96e-3e32-45a0-8648-647b2f58ab38" containerName="revoke-keys" Oct 31 00:52:41.160472 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.160351 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b6f96e-3e32-45a0-8648-647b2f58ab38" containerName="revoke-keys" Oct 31 00:52:41.160472 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.160412 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b6f96e-3e32-45a0-8648-647b2f58ab38" containerName="revoke-keys" Oct 31 00:52:41.164945 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.164926 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" Oct 31 00:52:41.170580 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.170549 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:52:41.170759 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.170697 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:52:41.170835 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.170764 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:52:41.171213 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.171089 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:52:41.171395 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.171375 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh"] Oct 31 00:52:41.331349 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.331313 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hhqrh\" (UID: \"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" Oct 31 00:52:41.331533 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.331385 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96clj\" (UniqueName: \"kubernetes.io/projected/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-kube-api-access-96clj\") pod \"maas-api-revoke-sub-79805ca9-hhqrh\" (UID: \"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" Oct 31 00:52:41.331533 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.331463 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hhqrh\" (UID: \"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" Oct 31 00:52:41.432382 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.432295 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96clj\" (UniqueName: \"kubernetes.io/projected/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-kube-api-access-96clj\") pod \"maas-api-revoke-sub-79805ca9-hhqrh\" (UID: \"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" Oct 31 00:52:41.432382 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.432355 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hhqrh\" (UID: \"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" Oct 31 00:52:41.432563 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.432406 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hhqrh\" (UID: \"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" Oct 31 00:52:41.433156 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.433131 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hhqrh\" (UID: \"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" Oct 31 00:52:41.434761 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.434745 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hhqrh\" (UID: \"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" Oct 31 00:52:41.441915 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.441892 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96clj\" (UniqueName: \"kubernetes.io/projected/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-kube-api-access-96clj\") pod \"maas-api-revoke-sub-79805ca9-hhqrh\" (UID: \"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" Oct 31 00:52:41.479410 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.479376 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" Oct 31 00:52:41.748438 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:41.748413 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh"] Oct 31 00:52:41.751146 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:52:41.751121 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1212f67_91d5_4bd0_8cc5_cc9e9fac383a.slice/crio-7d110a0f009d7e071a0fccd2e32eb29474f38734e2932431ea22ab31bdded727 WatchSource:0}: Error finding container 7d110a0f009d7e071a0fccd2e32eb29474f38734e2932431ea22ab31bdded727: Status 404 returned error can't find the container with id 7d110a0f009d7e071a0fccd2e32eb29474f38734e2932431ea22ab31bdded727 Oct 31 00:52:42.228968 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:42.228874 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" event={"ID":"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a","Type":"ContainerStarted","Data":"b4aef96d223500c54fb826992760cb806bab2156a8459ef216ee1becae496590"} Oct 31 00:52:42.228968 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:42.228925 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" event={"ID":"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a","Type":"ContainerStarted","Data":"7d110a0f009d7e071a0fccd2e32eb29474f38734e2932431ea22ab31bdded727"} Oct 31 00:52:42.248211 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:42.248163 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" podStartSLOduration=1.248149267 podStartE2EDuration="1.248149267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:42.246444935 +0000 UTC m=+1158.275625325" watchObservedRunningTime="2026-10-31 00:52:42.248149267 +0000 UTC m=+1158.277329697" Oct 31 00:52:43.234564 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:43.234527 4519 generic.go:356] "Generic (PLEG): container finished" podID="e1212f67-91d5-4bd0-8cc5-cc9e9fac383a" containerID="b4aef96d223500c54fb826992760cb806bab2156a8459ef216ee1becae496590" exitCode=0 Oct 31 00:52:43.234991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:43.234612 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" event={"ID":"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a","Type":"ContainerDied","Data":"b4aef96d223500c54fb826992760cb806bab2156a8459ef216ee1becae496590"} Oct 31 00:52:44.363822 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:44.363798 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" Oct 31 00:52:44.455777 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:44.455745 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-kube-api-access-96clj\" (UniqueName: \"kubernetes.io/projected/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-kube-api-access-96clj\") pod \"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a\" (UID: \"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a\") " Oct 31 00:52:44.455943 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:44.455806 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-maas-api-openshift-ca\") pod \"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a\" (UID: \"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a\") " Oct 31 00:52:44.455943 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:44.455920 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-maas-api-xks-ca\") pod \"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a\" (UID: \"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a\") " Oct 31 00:52:44.456177 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:44.456148 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-maas-api-openshift-ca" pod "e1212f67-91d5-4bd0-8cc5-cc9e9fac383a" (UID: "e1212f67-91d5-4bd0-8cc5-cc9e9fac383a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:44.457951 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:44.457932 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-maas-api-xks-ca" pod "e1212f67-91d5-4bd0-8cc5-cc9e9fac383a" (UID: "e1212f67-91d5-4bd0-8cc5-cc9e9fac383a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:52:44.458007 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:44.457961 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-kube-api-access-96clj" pod "e1212f67-91d5-4bd0-8cc5-cc9e9fac383a" (UID: "e1212f67-91d5-4bd0-8cc5-cc9e9fac383a"). InnerVolumeSpecName "kube-api-access-96clj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:44.556908 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:44.556833 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96clj\" (UniqueName: \"kubernetes.io/projected/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-kube-api-access-96clj\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:44.556908 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:44.556861 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:44.556908 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:44.556871 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:45.245295 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:45.245260 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" Oct 31 00:52:45.245486 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:45.245297 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh" event={"ID":"e1212f67-91d5-4bd0-8cc5-cc9e9fac383a","Type":"ContainerDied","Data":"7d110a0f009d7e071a0fccd2e32eb29474f38734e2932431ea22ab31bdded727"} Oct 31 00:52:45.245486 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:45.245349 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d110a0f009d7e071a0fccd2e32eb29474f38734e2932431ea22ab31bdded727" Oct 31 00:52:47.541498 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:52:47.541460 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:52:59.012947 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.012909 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj"] Oct 31 00:52:59.015509 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.013238 4519 cpu_manager.go:414] "containerMap: removing container" podUID="e1212f67-91d5-4bd0-8cc5-cc9e9fac383a" containerName="revoke-keys" Oct 31 00:52:59.015509 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.013249 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1212f67-91d5-4bd0-8cc5-cc9e9fac383a" containerName="revoke-keys" Oct 31 00:52:59.015509 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.013308 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1212f67-91d5-4bd0-8cc5-cc9e9fac383a" containerName="revoke-keys" Oct 31 00:52:59.016371 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.016354 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" Oct 31 00:52:59.023849 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.023809 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:52:59.023849 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.023824 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:52:59.024991 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.024963 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:52:59.025144 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.025129 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:52:59.031871 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.031836 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj"] Oct 31 00:52:59.118750 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.118710 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babe18a1-60d9-48d1-8f44-bb962f081c34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-w4pkj\" (UID: \"babe18a1-60d9-48d1-8f44-bb962f081c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" Oct 31 00:52:59.118750 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.118749 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djvjl\" (UniqueName: \"kubernetes.io/projected/babe18a1-60d9-48d1-8f44-bb962f081c34-kube-api-access-djvjl\") pod \"maas-api-revoke-sub-0b7e8edb-w4pkj\" (UID: \"babe18a1-60d9-48d1-8f44-bb962f081c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" Oct 31 00:52:59.118975 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.118889 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babe18a1-60d9-48d1-8f44-bb962f081c34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-w4pkj\" (UID: \"babe18a1-60d9-48d1-8f44-bb962f081c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" Oct 31 00:52:59.220179 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.220136 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babe18a1-60d9-48d1-8f44-bb962f081c34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-w4pkj\" (UID: \"babe18a1-60d9-48d1-8f44-bb962f081c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" Oct 31 00:52:59.220179 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.220177 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djvjl\" (UniqueName: \"kubernetes.io/projected/babe18a1-60d9-48d1-8f44-bb962f081c34-kube-api-access-djvjl\") pod \"maas-api-revoke-sub-0b7e8edb-w4pkj\" (UID: \"babe18a1-60d9-48d1-8f44-bb962f081c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" Oct 31 00:52:59.220408 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.220235 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babe18a1-60d9-48d1-8f44-bb962f081c34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-w4pkj\" (UID: \"babe18a1-60d9-48d1-8f44-bb962f081c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" Oct 31 00:52:59.220868 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.220842 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babe18a1-60d9-48d1-8f44-bb962f081c34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-w4pkj\" (UID: \"babe18a1-60d9-48d1-8f44-bb962f081c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" Oct 31 00:52:59.222503 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.222477 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babe18a1-60d9-48d1-8f44-bb962f081c34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-w4pkj\" (UID: \"babe18a1-60d9-48d1-8f44-bb962f081c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" Oct 31 00:52:59.228055 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.228022 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djvjl\" (UniqueName: \"kubernetes.io/projected/babe18a1-60d9-48d1-8f44-bb962f081c34-kube-api-access-djvjl\") pod \"maas-api-revoke-sub-0b7e8edb-w4pkj\" (UID: \"babe18a1-60d9-48d1-8f44-bb962f081c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" Oct 31 00:52:59.331905 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.331813 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" Oct 31 00:52:59.603150 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:52:59.603119 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj"] Oct 31 00:52:59.604554 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:52:59.604522 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabe18a1_60d9_48d1_8f44_bb962f081c34.slice/crio-42f86a33f651f7a6d3a5fa25397dfc7c4d82512da94beb91551da2a88631047c WatchSource:0}: Error finding container 42f86a33f651f7a6d3a5fa25397dfc7c4d82512da94beb91551da2a88631047c: Status 404 returned error can't find the container with id 42f86a33f651f7a6d3a5fa25397dfc7c4d82512da94beb91551da2a88631047c Oct 31 00:53:00.324263 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:00.324227 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" event={"ID":"babe18a1-60d9-48d1-8f44-bb962f081c34","Type":"ContainerStarted","Data":"21aa6caea6fed8afe33f0e90d38736608563244e870ab62aa420c14a88bae1aa"} Oct 31 00:53:00.324745 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:00.324280 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" event={"ID":"babe18a1-60d9-48d1-8f44-bb962f081c34","Type":"ContainerStarted","Data":"42f86a33f651f7a6d3a5fa25397dfc7c4d82512da94beb91551da2a88631047c"} Oct 31 00:53:00.343040 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:00.342986 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" podStartSLOduration=2.342971983 podStartE2EDuration="2.342971983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:00.34133877 +0000 UTC m=+1176.370519160" watchObservedRunningTime="2026-10-31 00:53:00.342971983 +0000 UTC m=+1176.372152372" Oct 31 00:53:01.330888 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:01.330853 4519 generic.go:356] "Generic (PLEG): container finished" podID="babe18a1-60d9-48d1-8f44-bb962f081c34" containerID="21aa6caea6fed8afe33f0e90d38736608563244e870ab62aa420c14a88bae1aa" exitCode=0 Oct 31 00:53:01.331290 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:01.330933 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" event={"ID":"babe18a1-60d9-48d1-8f44-bb962f081c34","Type":"ContainerDied","Data":"21aa6caea6fed8afe33f0e90d38736608563244e870ab62aa420c14a88bae1aa"} Oct 31 00:53:02.470464 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:02.470440 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" Oct 31 00:53:02.550000 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:02.549964 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/babe18a1-60d9-48d1-8f44-bb962f081c34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babe18a1-60d9-48d1-8f44-bb962f081c34-maas-api-xks-ca\") pod \"babe18a1-60d9-48d1-8f44-bb962f081c34\" (UID: \"babe18a1-60d9-48d1-8f44-bb962f081c34\") " Oct 31 00:53:02.550185 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:02.550033 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/babe18a1-60d9-48d1-8f44-bb962f081c34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babe18a1-60d9-48d1-8f44-bb962f081c34-maas-api-openshift-ca\") pod \"babe18a1-60d9-48d1-8f44-bb962f081c34\" (UID: \"babe18a1-60d9-48d1-8f44-bb962f081c34\") " Oct 31 00:53:02.550185 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:02.550062 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/babe18a1-60d9-48d1-8f44-bb962f081c34-kube-api-access-djvjl\" (UniqueName: \"kubernetes.io/projected/babe18a1-60d9-48d1-8f44-bb962f081c34-kube-api-access-djvjl\") pod \"babe18a1-60d9-48d1-8f44-bb962f081c34\" (UID: \"babe18a1-60d9-48d1-8f44-bb962f081c34\") " Oct 31 00:53:02.550385 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:02.550353 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babe18a1-60d9-48d1-8f44-bb962f081c34-maas-api-openshift-ca" pod "babe18a1-60d9-48d1-8f44-bb962f081c34" (UID: "babe18a1-60d9-48d1-8f44-bb962f081c34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:02.552058 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:02.552028 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/babe18a1-60d9-48d1-8f44-bb962f081c34-maas-api-xks-ca" pod "babe18a1-60d9-48d1-8f44-bb962f081c34" (UID: "babe18a1-60d9-48d1-8f44-bb962f081c34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:53:02.552150 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:02.552087 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babe18a1-60d9-48d1-8f44-bb962f081c34-kube-api-access-djvjl" pod "babe18a1-60d9-48d1-8f44-bb962f081c34" (UID: "babe18a1-60d9-48d1-8f44-bb962f081c34"). InnerVolumeSpecName "kube-api-access-djvjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:02.650818 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:02.650733 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djvjl\" (UniqueName: \"kubernetes.io/projected/babe18a1-60d9-48d1-8f44-bb962f081c34-kube-api-access-djvjl\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:02.650818 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:02.650767 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babe18a1-60d9-48d1-8f44-bb962f081c34-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:02.650818 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:02.650778 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babe18a1-60d9-48d1-8f44-bb962f081c34-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:03.341790 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:03.341759 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" Oct 31 00:53:03.341974 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:03.341760 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj" event={"ID":"babe18a1-60d9-48d1-8f44-bb962f081c34","Type":"ContainerDied","Data":"42f86a33f651f7a6d3a5fa25397dfc7c4d82512da94beb91551da2a88631047c"} Oct 31 00:53:03.341974 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:03.341877 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42f86a33f651f7a6d3a5fa25397dfc7c4d82512da94beb91551da2a88631047c" Oct 31 00:53:09.712867 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.712829 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf"] Oct 31 00:53:09.713263 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.713176 4519 cpu_manager.go:414] "containerMap: removing container" podUID="babe18a1-60d9-48d1-8f44-bb962f081c34" containerName="revoke-keys" Oct 31 00:53:09.713263 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.713190 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="babe18a1-60d9-48d1-8f44-bb962f081c34" containerName="revoke-keys" Oct 31 00:53:09.713263 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.713254 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="babe18a1-60d9-48d1-8f44-bb962f081c34" containerName="revoke-keys" Oct 31 00:53:09.716343 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.716327 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf" Oct 31 00:53:09.724917 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.724882 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:53:09.725323 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.725280 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:53:09.725445 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.725370 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:53:09.725630 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.725603 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:53:09.727929 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.727909 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf"] Oct 31 00:53:09.813334 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.813294 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4zflf\" (UID: \"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf" Oct 31 00:53:09.813548 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.813345 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4zflf\" (UID: \"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf" Oct 31 00:53:09.813548 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.813462 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2njz\" (UniqueName: \"kubernetes.io/projected/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-kube-api-access-l2njz\") pod \"maas-api-revoke-sub-3d4156f1-4zflf\" (UID: \"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf" Oct 31 00:53:09.914570 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.914525 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4zflf\" (UID: \"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf" Oct 31 00:53:09.914805 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.914584 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4zflf\" (UID: \"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf" Oct 31 00:53:09.914805 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.914664 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2njz\" (UniqueName: \"kubernetes.io/projected/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-kube-api-access-l2njz\") pod \"maas-api-revoke-sub-3d4156f1-4zflf\" (UID: \"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf" Oct 31 00:53:09.915295 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.915271 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4zflf\" (UID: \"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf" Oct 31 00:53:09.916994 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.916973 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4zflf\" (UID: \"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf" Oct 31 00:53:09.923196 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:09.923171 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2njz\" (UniqueName: \"kubernetes.io/projected/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-kube-api-access-l2njz\") pod \"maas-api-revoke-sub-3d4156f1-4zflf\" (UID: \"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf" Oct 31 00:53:10.030624 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:10.030586 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf" Oct 31 00:53:10.315164 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:10.315058 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf"] Oct 31 00:53:10.330739 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:53:10.330700 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4d721f_6c30_4708_bdb1_704a4e4f9a7e.slice/crio-5e42ba084891363d49b72b98d2c24fed39ab8a0c66e7863939d3d8c3f7d753f7 WatchSource:0}: Error finding container 5e42ba084891363d49b72b98d2c24fed39ab8a0c66e7863939d3d8c3f7d753f7: Status 404 returned error can't find the container with id 5e42ba084891363d49b72b98d2c24fed39ab8a0c66e7863939d3d8c3f7d753f7 Oct 31 00:53:10.374528 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:10.374491 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf" event={"ID":"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e","Type":"ContainerStarted","Data":"5e42ba084891363d49b72b98d2c24fed39ab8a0c66e7863939d3d8c3f7d753f7"} Oct 31 00:53:11.384192 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:11.384150 4519 generic.go:356] "Generic (PLEG): container finished" podID="ef4d721f-6c30-4708-bdb1-704a4e4f9a7e" containerID="8253bb22b3a1f6c1a8df546ac2a550f6dbc9ea03806227b656e42e9d5a140933" exitCode=0 Oct 31 00:53:11.384706 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:11.384234 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf" event={"ID":"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e","Type":"ContainerDied","Data":"8253bb22b3a1f6c1a8df546ac2a550f6dbc9ea03806227b656e42e9d5a140933"} Oct 31 00:53:11.769779 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:11.769742 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:12.517822 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:12.517797 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf" Oct 31 00:53:12.638137 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:12.638106 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-kube-api-access-l2njz\" (UniqueName: \"kubernetes.io/projected/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-kube-api-access-l2njz\") pod \"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e\" (UID: \"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e\") " Oct 31 00:53:12.638332 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:12.638185 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-maas-api-xks-ca\") pod \"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e\" (UID: \"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e\") " Oct 31 00:53:12.638332 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:12.638312 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-maas-api-openshift-ca\") pod \"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e\" (UID: \"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e\") " Oct 31 00:53:12.638720 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:12.638695 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-maas-api-openshift-ca" pod "ef4d721f-6c30-4708-bdb1-704a4e4f9a7e" (UID: "ef4d721f-6c30-4708-bdb1-704a4e4f9a7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:12.640220 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:12.640171 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-kube-api-access-l2njz" pod "ef4d721f-6c30-4708-bdb1-704a4e4f9a7e" (UID: "ef4d721f-6c30-4708-bdb1-704a4e4f9a7e"). InnerVolumeSpecName "kube-api-access-l2njz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:12.640295 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:12.640250 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-maas-api-xks-ca" pod "ef4d721f-6c30-4708-bdb1-704a4e4f9a7e" (UID: "ef4d721f-6c30-4708-bdb1-704a4e4f9a7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:53:12.739121 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:12.739083 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:12.739121 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:12.739115 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:12.739121 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:12.739126 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2njz\" (UniqueName: \"kubernetes.io/projected/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e-kube-api-access-l2njz\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:13.395023 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:13.394990 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf" Oct 31 00:53:13.395233 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:13.395025 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf" event={"ID":"ef4d721f-6c30-4708-bdb1-704a4e4f9a7e","Type":"ContainerDied","Data":"5e42ba084891363d49b72b98d2c24fed39ab8a0c66e7863939d3d8c3f7d753f7"} Oct 31 00:53:13.395233 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:13.395072 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e42ba084891363d49b72b98d2c24fed39ab8a0c66e7863939d3d8c3f7d753f7" Oct 31 00:53:17.504843 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:17.504810 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:17.540865 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:53:17.540827 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:53:18.701311 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:18.701280 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:25.812824 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:25.812784 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8wb6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:53:25.813243 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:25.812864 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8wb6m" podUID="a11bc6fe-fa8b-425e-b6df-007d3fc6aa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:53:29.735120 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:29.735081 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:29.842288 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:29.842251 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:30.928919 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:30.928884 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf"] Oct 31 00:53:30.935251 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:30.935225 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4zflf"] Oct 31 00:53:30.956176 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:30.956138 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m"] Oct 31 00:53:30.956889 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:30.956864 4519 cpu_manager.go:414] "containerMap: removing container" podUID="ef4d721f-6c30-4708-bdb1-704a4e4f9a7e" containerName="revoke-keys" Oct 31 00:53:30.957016 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:30.956890 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4d721f-6c30-4708-bdb1-704a4e4f9a7e" containerName="revoke-keys" Oct 31 00:53:30.957089 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:30.957074 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4d721f-6c30-4708-bdb1-704a4e4f9a7e" containerName="revoke-keys" Oct 31 00:53:30.961219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:30.961197 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m" Oct 31 00:53:30.964961 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:30.964930 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:53:30.965094 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:30.965051 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:53:30.965156 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:30.965097 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:53:30.965156 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:30.965136 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:53:30.965911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:30.965885 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m"] Oct 31 00:53:31.024368 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:31.024327 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw"] Oct 31 00:53:31.028782 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:31.028754 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41dbd2ef-gdxqw"] Oct 31 00:53:31.053627 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:31.053594 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdlq6\" (UniqueName: \"kubernetes.io/projected/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-kube-api-access-cdlq6\") pod \"maas-api-revoke-sub-3d4156f1-j829m\" (UID: \"1ebf3b5f-18a7-4a86-9e10-731e5de568c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m" Oct 31 00:53:31.053816 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:31.053654 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j829m\" (UID: \"1ebf3b5f-18a7-4a86-9e10-731e5de568c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m" Oct 31 00:53:31.053816 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:31.053763 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j829m\" (UID: \"1ebf3b5f-18a7-4a86-9e10-731e5de568c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m" Oct 31 00:53:31.154413 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:31.154376 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdlq6\" (UniqueName: \"kubernetes.io/projected/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-kube-api-access-cdlq6\") pod \"maas-api-revoke-sub-3d4156f1-j829m\" (UID: \"1ebf3b5f-18a7-4a86-9e10-731e5de568c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m" Oct 31 00:53:31.154413 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:31.154420 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j829m\" (UID: \"1ebf3b5f-18a7-4a86-9e10-731e5de568c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m" Oct 31 00:53:31.154704 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:31.154443 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j829m\" (UID: \"1ebf3b5f-18a7-4a86-9e10-731e5de568c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m" Oct 31 00:53:31.155073 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:31.155053 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j829m\" (UID: \"1ebf3b5f-18a7-4a86-9e10-731e5de568c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m" Oct 31 00:53:31.156801 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:31.156773 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j829m\" (UID: \"1ebf3b5f-18a7-4a86-9e10-731e5de568c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m" Oct 31 00:53:31.163184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:31.163159 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdlq6\" (UniqueName: \"kubernetes.io/projected/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-kube-api-access-cdlq6\") pod \"maas-api-revoke-sub-3d4156f1-j829m\" (UID: \"1ebf3b5f-18a7-4a86-9e10-731e5de568c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m" Oct 31 00:53:31.276844 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:31.276803 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m" Oct 31 00:53:31.550197 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:31.550167 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m"] Oct 31 00:53:31.551943 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:53:31.551910 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebf3b5f_18a7_4a86_9e10_731e5de568c9.slice/crio-177a4e56921f082e877f3b842a82be8384b9d0495cdf99420db8f35f497b33b0 WatchSource:0}: Error finding container 177a4e56921f082e877f3b842a82be8384b9d0495cdf99420db8f35f497b33b0: Status 404 returned error can't find the container with id 177a4e56921f082e877f3b842a82be8384b9d0495cdf99420db8f35f497b33b0 Oct 31 00:53:32.100230 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:32.100153 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f" path="/var/lib/kubelet/pods/ebb01356-7f9b-4d24-9f36-f4ad1fa75e5f/volumes" Oct 31 00:53:32.100603 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:32.100572 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef4d721f-6c30-4708-bdb1-704a4e4f9a7e" path="/var/lib/kubelet/pods/ef4d721f-6c30-4708-bdb1-704a4e4f9a7e/volumes" Oct 31 00:53:32.538684 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:32.538652 4519 generic.go:356] "Generic (PLEG): container finished" podID="1ebf3b5f-18a7-4a86-9e10-731e5de568c9" containerID="12efcdd749a03bc4071ef2d016e8d28fbcdd391d3d3934b418485691642aa63f" exitCode=0 Oct 31 00:53:32.538864 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:32.538735 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m" event={"ID":"1ebf3b5f-18a7-4a86-9e10-731e5de568c9","Type":"ContainerDied","Data":"12efcdd749a03bc4071ef2d016e8d28fbcdd391d3d3934b418485691642aa63f"} Oct 31 00:53:32.538864 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:32.538795 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m" event={"ID":"1ebf3b5f-18a7-4a86-9e10-731e5de568c9","Type":"ContainerStarted","Data":"177a4e56921f082e877f3b842a82be8384b9d0495cdf99420db8f35f497b33b0"} Oct 31 00:53:33.665197 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:33.665173 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m" Oct 31 00:53:33.674547 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:33.674526 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-maas-api-xks-ca\") pod \"1ebf3b5f-18a7-4a86-9e10-731e5de568c9\" (UID: \"1ebf3b5f-18a7-4a86-9e10-731e5de568c9\") " Oct 31 00:53:33.674680 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:33.674586 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-maas-api-openshift-ca\") pod \"1ebf3b5f-18a7-4a86-9e10-731e5de568c9\" (UID: \"1ebf3b5f-18a7-4a86-9e10-731e5de568c9\") " Oct 31 00:53:33.674680 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:33.674623 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-kube-api-access-cdlq6\" (UniqueName: \"kubernetes.io/projected/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-kube-api-access-cdlq6\") pod \"1ebf3b5f-18a7-4a86-9e10-731e5de568c9\" (UID: \"1ebf3b5f-18a7-4a86-9e10-731e5de568c9\") " Oct 31 00:53:33.674910 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:33.674883 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-maas-api-openshift-ca" pod "1ebf3b5f-18a7-4a86-9e10-731e5de568c9" (UID: "1ebf3b5f-18a7-4a86-9e10-731e5de568c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:33.676649 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:33.676610 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-kube-api-access-cdlq6" pod "1ebf3b5f-18a7-4a86-9e10-731e5de568c9" (UID: "1ebf3b5f-18a7-4a86-9e10-731e5de568c9"). InnerVolumeSpecName "kube-api-access-cdlq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:33.676752 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:33.676616 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-maas-api-xks-ca" pod "1ebf3b5f-18a7-4a86-9e10-731e5de568c9" (UID: "1ebf3b5f-18a7-4a86-9e10-731e5de568c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:53:33.776135 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:33.776104 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdlq6\" (UniqueName: \"kubernetes.io/projected/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-kube-api-access-cdlq6\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:33.776135 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:33.776131 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:33.776135 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:33.776142 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebf3b5f-18a7-4a86-9e10-731e5de568c9-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:34.549020 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:34.548992 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m" Oct 31 00:53:34.549173 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:34.548989 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m" event={"ID":"1ebf3b5f-18a7-4a86-9e10-731e5de568c9","Type":"ContainerDied","Data":"177a4e56921f082e877f3b842a82be8384b9d0495cdf99420db8f35f497b33b0"} Oct 31 00:53:34.549173 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:34.549131 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="177a4e56921f082e877f3b842a82be8384b9d0495cdf99420db8f35f497b33b0" Oct 31 00:53:44.022309 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:44.022225 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2"] Oct 31 00:53:44.026203 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:44.026172 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a90c24bc-gpsg2"] Oct 31 00:53:44.100508 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:44.100463 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a10862b6-342b-4eb3-9c1a-5ed66b45ba02" path="/var/lib/kubelet/pods/a10862b6-342b-4eb3-9c1a-5ed66b45ba02/volumes" Oct 31 00:53:47.541144 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:53:47.541110 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:53:54.372821 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:54.372769 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-457nv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:53:54.373504 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:54.373464 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" podUID="23d1c38d-9d1d-4674-82cc-38782f2a4f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:53:55.096817 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:55.096781 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ggdf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:53:55.097040 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:55.096833 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ggdf4" podUID="1a6c37d5-3025-470f-94c0-ee1502fc7a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:53:59.028469 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.028433 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8"] Oct 31 00:53:59.030148 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.030123 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742129c9-lvtv8"] Oct 31 00:53:59.642494 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.642458 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn"] Oct 31 00:53:59.642840 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.642821 4519 cpu_manager.go:414] "containerMap: removing container" podUID="1ebf3b5f-18a7-4a86-9e10-731e5de568c9" containerName="revoke-keys" Oct 31 00:53:59.642893 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.642842 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ebf3b5f-18a7-4a86-9e10-731e5de568c9" containerName="revoke-keys" Oct 31 00:53:59.642929 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.642908 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ebf3b5f-18a7-4a86-9e10-731e5de568c9" containerName="revoke-keys" Oct 31 00:53:59.646076 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.646059 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" Oct 31 00:53:59.654955 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.654923 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:53:59.655151 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.654934 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:53:59.655231 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.655016 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:53:59.655303 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.655116 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:53:59.656091 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.656064 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn"] Oct 31 00:53:59.744057 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.744017 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sljzn\" (UID: \"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" Oct 31 00:53:59.744275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.744079 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2df9n\" (UniqueName: \"kubernetes.io/projected/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-kube-api-access-2df9n\") pod \"maas-api-revoke-sub-e2fc9c5b-sljzn\" (UID: \"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" Oct 31 00:53:59.744357 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.744267 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sljzn\" (UID: \"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" Oct 31 00:53:59.845322 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.845290 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2df9n\" (UniqueName: \"kubernetes.io/projected/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-kube-api-access-2df9n\") pod \"maas-api-revoke-sub-e2fc9c5b-sljzn\" (UID: \"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" Oct 31 00:53:59.845504 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.845363 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sljzn\" (UID: \"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" Oct 31 00:53:59.845504 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.845411 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sljzn\" (UID: \"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" Oct 31 00:53:59.846049 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.846030 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sljzn\" (UID: \"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" Oct 31 00:53:59.848058 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.848037 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-sljzn\" (UID: \"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" Oct 31 00:53:59.854132 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.854106 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2df9n\" (UniqueName: \"kubernetes.io/projected/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-kube-api-access-2df9n\") pod \"maas-api-revoke-sub-e2fc9c5b-sljzn\" (UID: \"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" Oct 31 00:53:59.960823 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:53:59.960716 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" Oct 31 00:54:00.101030 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:00.100997 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="234959cb-ee8c-4761-9032-ec19e95d9106" path="/var/lib/kubelet/pods/234959cb-ee8c-4761-9032-ec19e95d9106/volumes" Oct 31 00:54:00.233873 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:00.233847 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn"] Oct 31 00:54:00.235954 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:54:00.235912 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2028f40f_a603_4ba9_bf1b_a3c0e95f1aab.slice/crio-fcdde0de3143e56a62b0b4deda9024df2f4a374fc8eb1f8c1d80a6f36dce2305 WatchSource:0}: Error finding container fcdde0de3143e56a62b0b4deda9024df2f4a374fc8eb1f8c1d80a6f36dce2305: Status 404 returned error can't find the container with id fcdde0de3143e56a62b0b4deda9024df2f4a374fc8eb1f8c1d80a6f36dce2305 Oct 31 00:54:00.713517 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:00.713414 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" event={"ID":"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab","Type":"ContainerStarted","Data":"a22e10eae35a8da38a266d6a5ce3c8745201a0dcab2dbb946bc206f3beead99b"} Oct 31 00:54:00.713517 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:00.713488 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" event={"ID":"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab","Type":"ContainerStarted","Data":"fcdde0de3143e56a62b0b4deda9024df2f4a374fc8eb1f8c1d80a6f36dce2305"} Oct 31 00:54:00.733061 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:00.733004 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" podStartSLOduration=1.732989031 podStartE2EDuration="1.732989031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:00.731079262 +0000 UTC m=+1236.760259675" watchObservedRunningTime="2026-10-31 00:54:00.732989031 +0000 UTC m=+1236.762169462" Oct 31 00:54:01.719722 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:01.719683 4519 generic.go:356] "Generic (PLEG): container finished" podID="2028f40f-a603-4ba9-bf1b-a3c0e95f1aab" containerID="a22e10eae35a8da38a266d6a5ce3c8745201a0dcab2dbb946bc206f3beead99b" exitCode=0 Oct 31 00:54:01.720147 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:01.719768 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" event={"ID":"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab","Type":"ContainerDied","Data":"a22e10eae35a8da38a266d6a5ce3c8745201a0dcab2dbb946bc206f3beead99b"} Oct 31 00:54:02.852997 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:02.852969 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" Oct 31 00:54:02.873200 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:02.873173 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-kube-api-access-2df9n\" (UniqueName: \"kubernetes.io/projected/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-kube-api-access-2df9n\") pod \"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab\" (UID: \"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab\") " Oct 31 00:54:02.873402 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:02.873216 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-maas-api-openshift-ca\") pod \"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab\" (UID: \"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab\") " Oct 31 00:54:02.873758 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:02.873725 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-maas-api-openshift-ca" pod "2028f40f-a603-4ba9-bf1b-a3c0e95f1aab" (UID: "2028f40f-a603-4ba9-bf1b-a3c0e95f1aab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:02.875722 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:02.875699 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-kube-api-access-2df9n" pod "2028f40f-a603-4ba9-bf1b-a3c0e95f1aab" (UID: "2028f40f-a603-4ba9-bf1b-a3c0e95f1aab"). InnerVolumeSpecName "kube-api-access-2df9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:02.973830 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:02.973793 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-maas-api-xks-ca\") pod \"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab\" (UID: \"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab\") " Oct 31 00:54:02.974008 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:02.973946 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2df9n\" (UniqueName: \"kubernetes.io/projected/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-kube-api-access-2df9n\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:02.974008 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:02.973957 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:02.975943 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:02.975889 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-maas-api-xks-ca" pod "2028f40f-a603-4ba9-bf1b-a3c0e95f1aab" (UID: "2028f40f-a603-4ba9-bf1b-a3c0e95f1aab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:03.074395 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:03.074357 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:03.730497 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:03.730460 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" event={"ID":"2028f40f-a603-4ba9-bf1b-a3c0e95f1aab","Type":"ContainerDied","Data":"fcdde0de3143e56a62b0b4deda9024df2f4a374fc8eb1f8c1d80a6f36dce2305"} Oct 31 00:54:03.730705 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:03.730513 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcdde0de3143e56a62b0b4deda9024df2f4a374fc8eb1f8c1d80a6f36dce2305" Oct 31 00:54:03.730705 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:03.730471 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn" Oct 31 00:54:09.866685 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:09.866630 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm"] Oct 31 00:54:09.867091 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:09.867005 4519 cpu_manager.go:414] "containerMap: removing container" podUID="2028f40f-a603-4ba9-bf1b-a3c0e95f1aab" containerName="revoke-keys" Oct 31 00:54:09.867091 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:09.867017 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2028f40f-a603-4ba9-bf1b-a3c0e95f1aab" containerName="revoke-keys" Oct 31 00:54:09.867168 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:09.867114 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2028f40f-a603-4ba9-bf1b-a3c0e95f1aab" containerName="revoke-keys" Oct 31 00:54:09.870309 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:09.870291 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" Oct 31 00:54:09.878211 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:09.878174 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:54:09.878446 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:09.878264 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:54:09.878446 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:09.878368 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:54:09.878446 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:09.878436 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:54:09.879822 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:09.879796 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm"] Oct 31 00:54:09.928534 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:09.928492 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25b67ad-adf1-447b-91e2-7d33e465a641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-vvcmm\" (UID: \"b25b67ad-adf1-447b-91e2-7d33e465a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" Oct 31 00:54:09.928534 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:09.928535 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25b67ad-adf1-447b-91e2-7d33e465a641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-vvcmm\" (UID: \"b25b67ad-adf1-447b-91e2-7d33e465a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" Oct 31 00:54:09.928815 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:09.928559 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwc6r\" (UniqueName: \"kubernetes.io/projected/b25b67ad-adf1-447b-91e2-7d33e465a641-kube-api-access-pwc6r\") pod \"maas-api-revoke-sub-5a29efb3-vvcmm\" (UID: \"b25b67ad-adf1-447b-91e2-7d33e465a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" Oct 31 00:54:10.029903 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:10.029855 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25b67ad-adf1-447b-91e2-7d33e465a641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-vvcmm\" (UID: \"b25b67ad-adf1-447b-91e2-7d33e465a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" Oct 31 00:54:10.029903 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:10.029906 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25b67ad-adf1-447b-91e2-7d33e465a641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-vvcmm\" (UID: \"b25b67ad-adf1-447b-91e2-7d33e465a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" Oct 31 00:54:10.030186 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:10.029946 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwc6r\" (UniqueName: \"kubernetes.io/projected/b25b67ad-adf1-447b-91e2-7d33e465a641-kube-api-access-pwc6r\") pod \"maas-api-revoke-sub-5a29efb3-vvcmm\" (UID: \"b25b67ad-adf1-447b-91e2-7d33e465a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" Oct 31 00:54:10.030534 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:10.030506 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25b67ad-adf1-447b-91e2-7d33e465a641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-vvcmm\" (UID: \"b25b67ad-adf1-447b-91e2-7d33e465a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" Oct 31 00:54:10.032354 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:10.032333 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25b67ad-adf1-447b-91e2-7d33e465a641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-vvcmm\" (UID: \"b25b67ad-adf1-447b-91e2-7d33e465a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" Oct 31 00:54:10.038595 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:10.038574 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwc6r\" (UniqueName: \"kubernetes.io/projected/b25b67ad-adf1-447b-91e2-7d33e465a641-kube-api-access-pwc6r\") pod \"maas-api-revoke-sub-5a29efb3-vvcmm\" (UID: \"b25b67ad-adf1-447b-91e2-7d33e465a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" Oct 31 00:54:10.185629 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:10.185467 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" Oct 31 00:54:10.461229 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:10.461202 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm"] Oct 31 00:54:10.463175 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:54:10.463144 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25b67ad_adf1_447b_91e2_7d33e465a641.slice/crio-7ef0681f6832b0e9fe0ca40c1239e20c4b66ae035df4c77cf61e1463db910120 WatchSource:0}: Error finding container 7ef0681f6832b0e9fe0ca40c1239e20c4b66ae035df4c77cf61e1463db910120: Status 404 returned error can't find the container with id 7ef0681f6832b0e9fe0ca40c1239e20c4b66ae035df4c77cf61e1463db910120 Oct 31 00:54:10.765996 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:10.765957 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" event={"ID":"b25b67ad-adf1-447b-91e2-7d33e465a641","Type":"ContainerStarted","Data":"c686a347496ee0992bce8dec43054acd984410bb7fdf80b979de456db19047ae"} Oct 31 00:54:10.766179 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:10.766020 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" event={"ID":"b25b67ad-adf1-447b-91e2-7d33e465a641","Type":"ContainerStarted","Data":"7ef0681f6832b0e9fe0ca40c1239e20c4b66ae035df4c77cf61e1463db910120"} Oct 31 00:54:10.786132 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:10.786076 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" podStartSLOduration=1.7860600469999999 podStartE2EDuration="1.786060047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:10.783937802 +0000 UTC m=+1246.813118193" watchObservedRunningTime="2026-10-31 00:54:10.786060047 +0000 UTC m=+1246.815240468" Oct 31 00:54:11.797738 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:11.797692 4519 generic.go:356] "Generic (PLEG): container finished" podID="b25b67ad-adf1-447b-91e2-7d33e465a641" containerID="c686a347496ee0992bce8dec43054acd984410bb7fdf80b979de456db19047ae" exitCode=0 Oct 31 00:54:11.798455 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:11.798425 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" event={"ID":"b25b67ad-adf1-447b-91e2-7d33e465a641","Type":"ContainerDied","Data":"c686a347496ee0992bce8dec43054acd984410bb7fdf80b979de456db19047ae"} Oct 31 00:54:11.917612 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:11.917573 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:13.206272 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:13.205862 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" Oct 31 00:54:13.303997 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:13.303960 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25b67ad-adf1-447b-91e2-7d33e465a641-kube-api-access-pwc6r\" (UniqueName: \"kubernetes.io/projected/b25b67ad-adf1-447b-91e2-7d33e465a641-kube-api-access-pwc6r\") pod \"b25b67ad-adf1-447b-91e2-7d33e465a641\" (UID: \"b25b67ad-adf1-447b-91e2-7d33e465a641\") " Oct 31 00:54:13.304195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:13.304063 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b25b67ad-adf1-447b-91e2-7d33e465a641-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25b67ad-adf1-447b-91e2-7d33e465a641-maas-api-xks-ca\") pod \"b25b67ad-adf1-447b-91e2-7d33e465a641\" (UID: \"b25b67ad-adf1-447b-91e2-7d33e465a641\") " Oct 31 00:54:13.304195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:13.304123 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b25b67ad-adf1-447b-91e2-7d33e465a641-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25b67ad-adf1-447b-91e2-7d33e465a641-maas-api-openshift-ca\") pod \"b25b67ad-adf1-447b-91e2-7d33e465a641\" (UID: \"b25b67ad-adf1-447b-91e2-7d33e465a641\") " Oct 31 00:54:13.304545 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:13.304525 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25b67ad-adf1-447b-91e2-7d33e465a641-maas-api-openshift-ca" pod "b25b67ad-adf1-447b-91e2-7d33e465a641" (UID: "b25b67ad-adf1-447b-91e2-7d33e465a641"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:13.307025 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:13.306990 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25b67ad-adf1-447b-91e2-7d33e465a641-kube-api-access-pwc6r" pod "b25b67ad-adf1-447b-91e2-7d33e465a641" (UID: "b25b67ad-adf1-447b-91e2-7d33e465a641"). InnerVolumeSpecName "kube-api-access-pwc6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:13.307315 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:13.307287 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25b67ad-adf1-447b-91e2-7d33e465a641-maas-api-xks-ca" pod "b25b67ad-adf1-447b-91e2-7d33e465a641" (UID: "b25b67ad-adf1-447b-91e2-7d33e465a641"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:13.404801 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:13.404704 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25b67ad-adf1-447b-91e2-7d33e465a641-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:13.404801 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:13.404739 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwc6r\" (UniqueName: \"kubernetes.io/projected/b25b67ad-adf1-447b-91e2-7d33e465a641-kube-api-access-pwc6r\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:13.404801 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:13.404754 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25b67ad-adf1-447b-91e2-7d33e465a641-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:13.813278 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:13.813248 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" Oct 31 00:54:13.813497 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:13.813288 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm" event={"ID":"b25b67ad-adf1-447b-91e2-7d33e465a641","Type":"ContainerDied","Data":"7ef0681f6832b0e9fe0ca40c1239e20c4b66ae035df4c77cf61e1463db910120"} Oct 31 00:54:13.813497 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:13.813348 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ef0681f6832b0e9fe0ca40c1239e20c4b66ae035df4c77cf61e1463db910120" Oct 31 00:54:15.103553 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:15.103515 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:17.540426 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:54:17.540386 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:54:25.652320 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:25.652276 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m"] Oct 31 00:54:25.673697 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:25.673663 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j829m"] Oct 31 00:54:26.067018 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:26.066984 4519 scope.go:122] "RemoveContainer" containerID="b818b342ff3e6653b1abfd7a1dfc5119472d7e39eeaa8c7c2feececf79a97799" Oct 31 00:54:26.075777 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:26.075755 4519 scope.go:122] "RemoveContainer" containerID="fb53b0a9b9810d8a7ffd6a203d3afac072db3616f9ba39c2528821e3c2e0d301" Oct 31 00:54:26.084039 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:26.084015 4519 scope.go:122] "RemoveContainer" containerID="f02c07d768765b6e6b5ec0e565cd629843324ce035c36d623447af55c514a6f9" Oct 31 00:54:26.092271 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:26.092250 4519 scope.go:122] "RemoveContainer" containerID="6c566468af90ba3a6a293880cd06ccc55cd311be9e6b9712b6d5a375333b18aa" Oct 31 00:54:26.100589 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:26.100561 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ebf3b5f-18a7-4a86-9e10-731e5de568c9" path="/var/lib/kubelet/pods/1ebf3b5f-18a7-4a86-9e10-731e5de568c9/volumes" Oct 31 00:54:44.866294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:44.866258 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:45.022926 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:45.022889 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2"] Oct 31 00:54:45.025133 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:45.025109 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-5ghz2"] Oct 31 00:54:45.521703 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:45.521662 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:45.618592 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:45.618561 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:46.100403 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:46.100366 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e57ee0ab-0cb5-4441-a910-49530df6fdc0" path="/var/lib/kubelet/pods/e57ee0ab-0cb5-4441-a910-49530df6fdc0/volumes" Oct 31 00:54:47.543877 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:54:47.543839 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:54:49.830578 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:49.830541 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:49.996499 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:49.996467 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:50.963540 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:50.963487 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:51.023303 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.023266 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4"] Oct 31 00:54:51.023795 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.023776 4519 cpu_manager.go:414] "containerMap: removing container" podUID="b25b67ad-adf1-447b-91e2-7d33e465a641" containerName="revoke-keys" Oct 31 00:54:51.023847 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.023799 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25b67ad-adf1-447b-91e2-7d33e465a641" containerName="revoke-keys" Oct 31 00:54:51.023937 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.023925 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25b67ad-adf1-447b-91e2-7d33e465a641" containerName="revoke-keys" Oct 31 00:54:51.027420 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.027396 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" Oct 31 00:54:51.036579 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.036547 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4"] Oct 31 00:54:51.043568 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.043519 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-5d01f878-dockercfg-sdb69\"" Oct 31 00:54:51.070950 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.070893 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:51.090255 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.090222 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqxvt\" (UniqueName: \"kubernetes.io/projected/d46149f1-da3b-4442-bdd4-129a16c73b57-kube-api-access-hqxvt\") pod \"payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4\" (UID: \"d46149f1-da3b-4442-bdd4-129a16c73b57\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" Oct 31 00:54:51.090396 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.090295 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d46149f1-da3b-4442-bdd4-129a16c73b57-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4\" (UID: \"d46149f1-da3b-4442-bdd4-129a16c73b57\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" Oct 31 00:54:51.113814 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.113782 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-fb32d0\"" Oct 31 00:54:51.172227 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.172188 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:51.191221 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.191179 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d46149f1-da3b-4442-bdd4-129a16c73b57-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4\" (UID: \"d46149f1-da3b-4442-bdd4-129a16c73b57\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" Oct 31 00:54:51.191418 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.191283 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqxvt\" (UniqueName: \"kubernetes.io/projected/d46149f1-da3b-4442-bdd4-129a16c73b57-kube-api-access-hqxvt\") pod \"payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4\" (UID: \"d46149f1-da3b-4442-bdd4-129a16c73b57\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" Oct 31 00:54:51.191955 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.191935 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d46149f1-da3b-4442-bdd4-129a16c73b57-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4\" (UID: \"d46149f1-da3b-4442-bdd4-129a16c73b57\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" Oct 31 00:54:51.199582 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.199554 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqxvt\" (UniqueName: \"kubernetes.io/projected/d46149f1-da3b-4442-bdd4-129a16c73b57-kube-api-access-hqxvt\") pod \"payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4\" (UID: \"d46149f1-da3b-4442-bdd4-129a16c73b57\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" Oct 31 00:54:51.341920 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.341878 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" Oct 31 00:54:51.640022 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.639981 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4"] Oct 31 00:54:51.642315 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:54:51.642283 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46149f1_da3b_4442_bdd4_129a16c73b57.slice/crio-0bf0ecd82fef9202ff1b6b9604a2580af85d860bb8665968de369209263b68ec WatchSource:0}: Error finding container 0bf0ecd82fef9202ff1b6b9604a2580af85d860bb8665968de369209263b68ec: Status 404 returned error can't find the container with id 0bf0ecd82fef9202ff1b6b9604a2580af85d860bb8665968de369209263b68ec Oct 31 00:54:51.790887 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:51.790852 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:52.031569 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:52.031531 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" event={"ID":"d46149f1-da3b-4442-bdd4-129a16c73b57","Type":"ContainerStarted","Data":"b43aac7bc15495e7a715780f665cdb0667bc35a4371778b75d7f4e51e5175559"} Oct 31 00:54:52.032029 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:52.031581 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" event={"ID":"d46149f1-da3b-4442-bdd4-129a16c73b57","Type":"ContainerStarted","Data":"0bf0ecd82fef9202ff1b6b9604a2580af85d860bb8665968de369209263b68ec"} Oct 31 00:54:52.032029 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:52.031602 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" Oct 31 00:54:52.081750 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:52.077713 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" podStartSLOduration=1.077691226 podStartE2EDuration="1.077691226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:52.074433 +0000 UTC m=+1288.103613391" watchObservedRunningTime="2026-10-31 00:54:52.077691226 +0000 UTC m=+1288.106871617" Oct 31 00:54:52.463734 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:52.463610 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:52.602922 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:52.602876 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:53.246566 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:53.246536 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:54.047168 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:54.047130 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:54.752211 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:54.752176 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:55.375836 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:55.375804 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:55.976365 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:55.976327 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:56.646421 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:56.646384 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:57.258145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:57.258113 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:57.858377 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:57.858332 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:58.630975 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:58.630942 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:59.245479 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:59.245448 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:59.872618 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:59.872579 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:59.977443 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:54:59.977401 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:00.487521 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:00.487485 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:00.593046 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:00.593009 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:00.690457 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:00.690425 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:00.792271 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:00.792168 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:01.300003 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:01.299972 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:02.044471 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:02.044437 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:02.649057 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:02.649021 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:02.963258 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:02.963172 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch"] Oct 31 00:55:02.966888 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:02.966856 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:02.976983 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:02.976959 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fb32d0\"/\"openshift-service-ca.crt\"" Oct 31 00:55:02.979563 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:02.979546 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fb32d0\"/\"kube-root-ca.crt\"" Oct 31 00:55:02.979764 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:02.979748 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fb32d0\"/\"default-dockercfg-5gb5j\"" Oct 31 00:55:02.983005 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:02.982984 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fb32d0\"/\"simulator-single-main-fb32d0-kserve-self-signed-certs\"" Oct 31 00:55:02.987102 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:02.987072 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch"] Oct 31 00:55:03.010489 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.010456 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:03.021726 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.021694 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-dshm\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.021919 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.021777 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-tmp-dir\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.021963 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.021929 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/394705fd-8f0a-421c-83d6-e43d1fa69e4b-tls-certs\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.022001 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.021982 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-model-cache\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.022126 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.022100 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-home\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.022169 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.022151 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkjmg\" (UniqueName: \"kubernetes.io/projected/394705fd-8f0a-421c-83d6-e43d1fa69e4b-kube-api-access-zkjmg\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.041299 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.041269 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" Oct 31 00:55:03.123247 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.123213 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-tmp-dir\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.123430 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.123291 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/394705fd-8f0a-421c-83d6-e43d1fa69e4b-tls-certs\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.123430 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.123316 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-model-cache\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.123430 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.123335 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-home\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.123430 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.123364 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkjmg\" (UniqueName: \"kubernetes.io/projected/394705fd-8f0a-421c-83d6-e43d1fa69e4b-kube-api-access-zkjmg\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.123430 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.123401 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-dshm\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.124061 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.123776 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-model-cache\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.124061 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.123796 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-tmp-dir\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.124572 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.124549 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-home\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.126085 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.126061 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/394705fd-8f0a-421c-83d6-e43d1fa69e4b-tls-certs\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.126292 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.126274 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-dshm\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.135755 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.135726 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkjmg\" (UniqueName: \"kubernetes.io/projected/394705fd-8f0a-421c-83d6-e43d1fa69e4b-kube-api-access-zkjmg\") pod \"simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.299544 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.299503 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:03.437581 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.437544 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:03.574455 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:03.574426 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch"] Oct 31 00:55:03.576317 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:55:03.576290 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod394705fd_8f0a_421c_83d6_e43d1fa69e4b.slice/crio-ef41ce636232c2c9a20672e3bd3cd1b89c26585c9c3a3dce341bfcb062282f4c WatchSource:0}: Error finding container ef41ce636232c2c9a20672e3bd3cd1b89c26585c9c3a3dce341bfcb062282f4c: Status 404 returned error can't find the container with id ef41ce636232c2c9a20672e3bd3cd1b89c26585c9c3a3dce341bfcb062282f4c Oct 31 00:55:04.093603 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:04.093568 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" event={"ID":"394705fd-8f0a-421c-83d6-e43d1fa69e4b","Type":"ContainerStarted","Data":"d8f0cd032e48b9d4904378bbdb2fb21469f76c91931dde9ed523343d7cac06bb"} Oct 31 00:55:04.093603 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:04.093608 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" event={"ID":"394705fd-8f0a-421c-83d6-e43d1fa69e4b","Type":"ContainerStarted","Data":"ef41ce636232c2c9a20672e3bd3cd1b89c26585c9c3a3dce341bfcb062282f4c"} Oct 31 00:55:04.111929 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:04.111873 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" podStartSLOduration=2.111859114 podStartE2EDuration="2.111859114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:04.110734539 +0000 UTC m=+1300.139914932" watchObservedRunningTime="2026-10-31 00:55:04.111859114 +0000 UTC m=+1300.141039534" Oct 31 00:55:04.150416 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:04.150382 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:04.299629 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:04.299594 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:04.902701 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:04.901403 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:05.654773 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:05.654738 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:06.351986 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:06.351943 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:07.041580 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:07.041546 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:07.733812 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:07.733780 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:08.433535 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:08.433500 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:08.941066 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:08.941031 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:09.565173 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:09.565138 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:10.242899 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:10.242868 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:10.864864 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:10.864826 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:11.558965 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:11.558926 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:11.948415 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:11.948329 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:12.114525 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:12.114490 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:12.215366 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:12.215281 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:12.861168 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:12.861133 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:13.300071 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:13.300032 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:13.312653 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:13.312602 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:13.454680 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:13.454646 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:14.051327 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:14.051292 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:14.152028 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:14.151995 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:55:14.742430 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:14.742393 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:15.359958 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:15.359911 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:15.971399 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:15.971363 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:16.553265 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:16.553232 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:17.159826 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:17.159792 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:17.542179 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:55:17.542140 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:55:17.755110 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:17.755077 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:18.438197 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:18.438163 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:19.050208 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:19.050174 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:19.667129 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:19.667096 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:20.363781 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:20.363739 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:21.047381 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:21.047349 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:21.651012 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:21.650979 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:22.310937 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:22.310900 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:23.053485 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:23.053423 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:23.649125 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:23.649092 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:24.244947 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:24.244910 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:24.849098 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:24.849062 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:25.455869 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:25.455835 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:26.049193 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:26.049153 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:26.130063 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:26.130029 4519 scope.go:122] "RemoveContainer" containerID="2edd1c505c031de3290b4febe911e89cbee0c8ca4605b95f6fd30ef7f4253e34" Oct 31 00:55:26.138738 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:26.138718 4519 scope.go:122] "RemoveContainer" containerID="4c3a3dc9dca290beeaaf3e31c75798d9b3951c6ff8c37e3964934cc8ed5cc71a" Oct 31 00:55:26.146814 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:26.146788 4519 scope.go:122] "RemoveContainer" containerID="f971c321ea2f0fb242f5b6c007cf5bf3ed3b6e61c409e0f807d8445c5968ef3a" Oct 31 00:55:26.754384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:26.754354 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:27.426999 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:27.426945 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:27.957063 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:27.957027 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:28.652291 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:28.652253 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:29.256768 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:29.256738 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:29.907826 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:29.907791 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:30.548708 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:30.548668 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:31.167733 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:31.167699 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:31.847842 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:31.847810 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:32.575920 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:32.575836 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:33.259842 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:33.259806 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:33.998645 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:33.998596 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:34.539500 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:34.539468 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:35.153277 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:35.153244 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:35.746280 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:35.746248 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:36.355877 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:36.355844 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:37.056212 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:37.056161 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:37.641231 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:37.641199 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:38.257941 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:38.257905 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:38.850155 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:38.850120 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:39.450048 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:39.450010 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:40.048320 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:40.048284 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:40.653518 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:40.653474 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:41.051513 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:41.051479 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw"] Oct 31 00:55:41.057153 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:41.057125 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tfxbw"] Oct 31 00:55:41.364384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:41.364298 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:41.950903 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:41.950868 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:42.101103 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:42.101067 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e6cf90-2c26-4881-bd77-798d533b7dd3" path="/var/lib/kubelet/pods/54e6cf90-2c26-4881-bd77-798d533b7dd3/volumes" Oct 31 00:55:42.603743 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:42.603708 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:42.974897 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:42.974809 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:43.210918 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:43.210879 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:43.984326 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:43.984291 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:44.770411 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:44.770361 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:45.458384 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:45.458349 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:46.114899 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:46.114861 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:46.805622 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:46.805588 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:47.457177 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:47.457142 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:47.556025 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:55:47.555988 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:55:47.940689 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:47.940655 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch"] Oct 31 00:55:48.036705 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:48.036672 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:48.152381 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:48.152345 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:48.752553 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:48.752522 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:49.337893 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:49.337860 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:49.944318 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:49.944288 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:50.540067 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:50.540030 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:51.140685 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:51.140625 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:51.743425 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:51.743392 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:52.392548 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:52.392516 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:53.127867 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:53.127833 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:53.334724 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:53.334684 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4"] Oct 31 00:55:53.382367 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:53.382266 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:53.483763 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:53.483734 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:53.651227 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:53.651143 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:53.851871 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:53.851836 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:53.921180 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:53.921093 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:54.396842 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:54.396814 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:58.335289 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:58.335214 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" podUID="d46149f1-da3b-4442-bdd4-129a16c73b57" containerName="payload-pre-processing" containerID="cri-o://b43aac7bc15495e7a715780f665cdb0667bc35a4371778b75d7f4e51e5175559" gracePeriod=25 Oct 31 00:55:59.023303 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:59.023267 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8"] Oct 31 00:55:59.025750 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:55:59.025721 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jmvm8"] Oct 31 00:56:00.025606 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:00.025566 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:00.097311 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:00.097277 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:00.117550 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:00.116417 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56dce295-cd27-432a-aa4f-4e86ff4d070b" path="/var/lib/kubelet/pods/56dce295-cd27-432a-aa4f-4e86ff4d070b/volumes" Oct 31 00:56:00.403463 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:00.403384 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:02.960294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:02.960253 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" podUID="394705fd-8f0a-421c-83d6-e43d1fa69e4b" containerName="main" containerID="cri-o://d8f0cd032e48b9d4904378bbdb2fb21469f76c91931dde9ed523343d7cac06bb" gracePeriod=45 Oct 31 00:56:03.039191 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.039156 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" podUID="d46149f1-da3b-4442-bdd4-129a16c73b57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:56:03.196334 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.196311 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:56:03.327500 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.327470 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/394705fd-8f0a-421c-83d6-e43d1fa69e4b-tls-certs\" (UniqueName: \"kubernetes.io/secret/394705fd-8f0a-421c-83d6-e43d1fa69e4b-tls-certs\") pod \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " Oct 31 00:56:03.327722 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.327527 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-home\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-home\") pod \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " Oct 31 00:56:03.327722 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.327598 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394705fd-8f0a-421c-83d6-e43d1fa69e4b-kube-api-access-zkjmg\" (UniqueName: \"kubernetes.io/projected/394705fd-8f0a-421c-83d6-e43d1fa69e4b-kube-api-access-zkjmg\") pod \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " Oct 31 00:56:03.327722 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.327655 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-dshm\") pod \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " Oct 31 00:56:03.327905 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.327728 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-model-cache\") pod \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " Oct 31 00:56:03.327905 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.327773 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-tmp-dir\") pod \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\" (UID: \"394705fd-8f0a-421c-83d6-e43d1fa69e4b\") " Oct 31 00:56:03.327905 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.327878 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-home" pod "394705fd-8f0a-421c-83d6-e43d1fa69e4b" (UID: "394705fd-8f0a-421c-83d6-e43d1fa69e4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.328070 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.328000 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-model-cache" pod "394705fd-8f0a-421c-83d6-e43d1fa69e4b" (UID: "394705fd-8f0a-421c-83d6-e43d1fa69e4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.328123 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.328066 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-tmp-dir" pod "394705fd-8f0a-421c-83d6-e43d1fa69e4b" (UID: "394705fd-8f0a-421c-83d6-e43d1fa69e4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.328215 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.328198 4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-home\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.328280 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.328217 4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-model-cache\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.328280 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.328227 4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-tmp-dir\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.329920 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.329893 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-dshm" pod "394705fd-8f0a-421c-83d6-e43d1fa69e4b" (UID: "394705fd-8f0a-421c-83d6-e43d1fa69e4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.330050 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.330023 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394705fd-8f0a-421c-83d6-e43d1fa69e4b-tls-certs" pod "394705fd-8f0a-421c-83d6-e43d1fa69e4b" (UID: "394705fd-8f0a-421c-83d6-e43d1fa69e4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:03.330184 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.330160 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394705fd-8f0a-421c-83d6-e43d1fa69e4b-kube-api-access-zkjmg" pod "394705fd-8f0a-421c-83d6-e43d1fa69e4b" (UID: "394705fd-8f0a-421c-83d6-e43d1fa69e4b"). InnerVolumeSpecName "kube-api-access-zkjmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:03.384571 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.384539 4519 generic.go:356] "Generic (PLEG): container finished" podID="394705fd-8f0a-421c-83d6-e43d1fa69e4b" containerID="d8f0cd032e48b9d4904378bbdb2fb21469f76c91931dde9ed523343d7cac06bb" exitCode=0 Oct 31 00:56:03.384774 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.384591 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" Oct 31 00:56:03.384774 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.384590 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" event={"ID":"394705fd-8f0a-421c-83d6-e43d1fa69e4b","Type":"ContainerDied","Data":"d8f0cd032e48b9d4904378bbdb2fb21469f76c91931dde9ed523343d7cac06bb"} Oct 31 00:56:03.384774 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.384676 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" event={"ID":"394705fd-8f0a-421c-83d6-e43d1fa69e4b","Type":"ContainerDied","Data":"ef41ce636232c2c9a20672e3bd3cd1b89c26585c9c3a3dce341bfcb062282f4c"} Oct 31 00:56:03.384774 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.384694 4519 scope.go:122] "RemoveContainer" containerID="d8f0cd032e48b9d4904378bbdb2fb21469f76c91931dde9ed523343d7cac06bb" Oct 31 00:56:03.393668 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.393647 4519 scope.go:122] "RemoveContainer" containerID="d8f0cd032e48b9d4904378bbdb2fb21469f76c91931dde9ed523343d7cac06bb" Oct 31 00:56:03.393924 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:56:03.393901 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8f0cd032e48b9d4904378bbdb2fb21469f76c91931dde9ed523343d7cac06bb\": container with ID starting with d8f0cd032e48b9d4904378bbdb2fb21469f76c91931dde9ed523343d7cac06bb not found: ID does not exist" containerID="d8f0cd032e48b9d4904378bbdb2fb21469f76c91931dde9ed523343d7cac06bb" Oct 31 00:56:03.393989 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.393937 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f0cd032e48b9d4904378bbdb2fb21469f76c91931dde9ed523343d7cac06bb"} err="failed to get container status \"d8f0cd032e48b9d4904378bbdb2fb21469f76c91931dde9ed523343d7cac06bb\": rpc error: code = NotFound desc = could not find container \"d8f0cd032e48b9d4904378bbdb2fb21469f76c91931dde9ed523343d7cac06bb\": container with ID starting with d8f0cd032e48b9d4904378bbdb2fb21469f76c91931dde9ed523343d7cac06bb not found: ID does not exist" Oct 31 00:56:03.407187 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.407164 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch"] Oct 31 00:56:03.408679 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.408660 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch"] Oct 31 00:56:03.428924 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.428900 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkjmg\" (UniqueName: \"kubernetes.io/projected/394705fd-8f0a-421c-83d6-e43d1fa69e4b-kube-api-access-zkjmg\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.428924 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.428925 4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/394705fd-8f0a-421c-83d6-e43d1fa69e4b-dshm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.429033 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:03.428934 4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/394705fd-8f0a-421c-83d6-e43d1fa69e4b-tls-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:04.106165 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:04.106130 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394705fd-8f0a-421c-83d6-e43d1fa69e4b" path="/var/lib/kubelet/pods/394705fd-8f0a-421c-83d6-e43d1fa69e4b/volumes" Oct 31 00:56:04.142005 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:04.141966 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-fb32d0/simulator-single-main-fb32d0-kserve-7db7d6fc99-tv8ch" podUID="394705fd-8f0a-421c-83d6-e43d1fa69e4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 31 00:56:11.023943 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:11.023908 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6"] Oct 31 00:56:11.028120 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:11.028092 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4j9h6"] Oct 31 00:56:11.981749 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:11.981709 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:12.100653 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:12.100607 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98b5761b-cdcb-430f-bf12-899eb68a11d9" path="/var/lib/kubelet/pods/98b5761b-cdcb-430f-bf12-899eb68a11d9/volumes" Oct 31 00:56:12.149289 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:12.149255 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:13.056661 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:13.054737 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" podUID="d46149f1-da3b-4442-bdd4-129a16c73b57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:56:17.542354 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:56:17.542317 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:56:18.487028 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:18.486984 4519 generic.go:356] "Generic (PLEG): container finished" podID="d46149f1-da3b-4442-bdd4-129a16c73b57" containerID="b43aac7bc15495e7a715780f665cdb0667bc35a4371778b75d7f4e51e5175559" exitCode=0 Oct 31 00:56:18.487028 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:18.487019 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" event={"ID":"d46149f1-da3b-4442-bdd4-129a16c73b57","Type":"ContainerDied","Data":"b43aac7bc15495e7a715780f665cdb0667bc35a4371778b75d7f4e51e5175559"} Oct 31 00:56:18.578092 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:18.578067 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" Oct 31 00:56:18.658260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:18.658177 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d46149f1-da3b-4442-bdd4-129a16c73b57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d46149f1-da3b-4442-bdd4-129a16c73b57-plugins-config-volume\") pod \"d46149f1-da3b-4442-bdd4-129a16c73b57\" (UID: \"d46149f1-da3b-4442-bdd4-129a16c73b57\") " Oct 31 00:56:18.658260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:18.658228 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46149f1-da3b-4442-bdd4-129a16c73b57-kube-api-access-hqxvt\" (UniqueName: \"kubernetes.io/projected/d46149f1-da3b-4442-bdd4-129a16c73b57-kube-api-access-hqxvt\") pod \"d46149f1-da3b-4442-bdd4-129a16c73b57\" (UID: \"d46149f1-da3b-4442-bdd4-129a16c73b57\") " Oct 31 00:56:18.658575 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:18.658551 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d46149f1-da3b-4442-bdd4-129a16c73b57-plugins-config-volume" pod "d46149f1-da3b-4442-bdd4-129a16c73b57" (UID: "d46149f1-da3b-4442-bdd4-129a16c73b57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:18.660412 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:18.660383 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46149f1-da3b-4442-bdd4-129a16c73b57-kube-api-access-hqxvt" pod "d46149f1-da3b-4442-bdd4-129a16c73b57" (UID: "d46149f1-da3b-4442-bdd4-129a16c73b57"). InnerVolumeSpecName "kube-api-access-hqxvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:18.758996 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:18.758942 4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d46149f1-da3b-4442-bdd4-129a16c73b57-plugins-config-volume\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:18.758996 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:18.758991 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqxvt\" (UniqueName: \"kubernetes.io/projected/d46149f1-da3b-4442-bdd4-129a16c73b57-kube-api-access-hqxvt\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:19.492801 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:19.492771 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" Oct 31 00:56:19.492993 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:19.492802 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4" event={"ID":"d46149f1-da3b-4442-bdd4-129a16c73b57","Type":"ContainerDied","Data":"0bf0ecd82fef9202ff1b6b9604a2580af85d860bb8665968de369209263b68ec"} Oct 31 00:56:19.492993 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:19.492858 4519 scope.go:122] "RemoveContainer" containerID="b43aac7bc15495e7a715780f665cdb0667bc35a4371778b75d7f4e51e5175559" Oct 31 00:56:19.515053 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:19.515026 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4"] Oct 31 00:56:19.518697 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:19.518671 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fb32d0-579bds42q4"] Oct 31 00:56:20.100586 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:20.100552 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d46149f1-da3b-4442-bdd4-129a16c73b57" path="/var/lib/kubelet/pods/d46149f1-da3b-4442-bdd4-129a16c73b57/volumes" Oct 31 00:56:24.783662 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:24.783609 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:25.034490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:25.034410 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:26.173690 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:26.173657 4519 scope.go:122] "RemoveContainer" containerID="934c4435861a803edaf7a81beb6435f5949519ad2d6c3f47ec3ae78abde56787" Oct 31 00:56:30.792037 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:30.792001 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:37.027689 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:37.027580 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks"] Oct 31 00:56:37.029111 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:37.029088 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9bmks"] Oct 31 00:56:38.101218 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:38.101180 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="273cc646-5b6e-4ab0-892d-c37d35ab4995" path="/var/lib/kubelet/pods/273cc646-5b6e-4ab0-892d-c37d35ab4995/volumes" Oct 31 00:56:43.719321 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:43.719284 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:47.023696 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:47.023658 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr"] Oct 31 00:56:47.025906 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:47.025883 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-x9rjr"] Oct 31 00:56:47.541141 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:56:47.541107 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:56:48.100385 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:56:48.100352 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51b515b8-c37e-4630-abe4-0cbdcbdbc084" path="/var/lib/kubelet/pods/51b515b8-c37e-4630-abe4-0cbdcbdbc084/volumes" Oct 31 00:57:08.032879 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:57:08.032840 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq"] Oct 31 00:57:08.035797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:57:08.035769 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nr7bq"] Oct 31 00:57:08.100881 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:57:08.100850 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf6b86a-2431-4443-90fe-56e2956b230e" path="/var/lib/kubelet/pods/8cf6b86a-2431-4443-90fe-56e2956b230e/volumes" Oct 31 00:57:12.859145 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:57:12.859109 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:12.959451 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:57:12.959415 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:17.541454 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:57:17.541409 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:57:26.022071 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:57:26.022030 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47"] Oct 31 00:57:26.025628 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:57:26.025596 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zrw47"] Oct 31 00:57:26.101089 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:57:26.101056 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b6f96e-3e32-45a0-8648-647b2f58ab38" path="/var/lib/kubelet/pods/f1b6f96e-3e32-45a0-8648-647b2f58ab38/volumes" Oct 31 00:57:26.202652 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:57:26.202611 4519 scope.go:122] "RemoveContainer" containerID="c10e8f2a70c4130331beb4dd18c718100f87251bb68cff1b8da54fd233e89e56" Oct 31 00:57:26.210919 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:57:26.210896 4519 scope.go:122] "RemoveContainer" containerID="b0c4fd71d53e386770d6094129160d9545aae598915b20ce0123578ea8356b83" Oct 31 00:57:26.218518 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:57:26.218492 4519 scope.go:122] "RemoveContainer" containerID="6d9c2e99cd1081a5ba4d4678858d479a127d7386d59eea74bfa6d4fc1e874e51" Oct 31 00:57:45.022244 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:57:45.022204 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh"] Oct 31 00:57:45.025001 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:57:45.024972 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hhqrh"] Oct 31 00:57:46.100461 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:57:46.100423 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1212f67-91d5-4bd0-8cc5-cc9e9fac383a" path="/var/lib/kubelet/pods/e1212f67-91d5-4bd0-8cc5-cc9e9fac383a/volumes" Oct 31 00:57:46.618679 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:57:46.618648 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:47.540446 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:57:47.540413 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:58:03.022923 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:03.022882 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj"] Oct 31 00:58:03.025480 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:03.025112 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-w4pkj"] Oct 31 00:58:04.100924 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:04.100893 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="babe18a1-60d9-48d1-8f44-bb962f081c34" path="/var/lib/kubelet/pods/babe18a1-60d9-48d1-8f44-bb962f081c34/volumes" Oct 31 00:58:12.075544 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:12.075455 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:15.701345 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:15.701309 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:16.112057 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:16.112012 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:16.210669 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:16.208506 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:17.541384 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:58:17.541346 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:58:19.841434 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:19.841386 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:20.427613 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:20.427578 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:21.003000 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:21.002969 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:21.095176 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:21.095143 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:21.174874 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:21.174833 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:21.850535 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:21.850500 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:22.581860 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:22.581825 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:23.365407 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:23.365372 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:24.056451 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:24.056414 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:24.751226 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:24.751182 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:25.504360 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:25.504325 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:26.148413 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:26.148366 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:26.244058 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:26.244026 4519 scope.go:122] "RemoveContainer" containerID="539644a855408e2bab4c10fc1398c1ff11ea1d8e8c65b82c4cd4e0ccfb62f3d4" Oct 31 00:58:26.251875 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:26.251853 4519 scope.go:122] "RemoveContainer" containerID="7ff15474d8056e1dd8158af12619b0c98c953bfb3e7ae9ce5ae6ebfc53fe6733" Oct 31 00:58:26.258993 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:26.258973 4519 scope.go:122] "RemoveContainer" containerID="de76857a481490370a11fc42c69a654b73e75f9ea81e0482b318166d7f58def9" Oct 31 00:58:26.266205 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:26.266182 4519 scope.go:122] "RemoveContainer" containerID="4e679e4922d0498f2b93ec8f9b253248dffcbec653a0119a005e0471f5920119" Oct 31 00:58:26.753904 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:26.753869 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:27.359852 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:27.359817 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:27.949075 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:27.949042 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:28.537300 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:28.537265 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:29.245371 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:29.245340 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:29.917908 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:29.917874 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:30.553589 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:30.553560 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:31.152369 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:31.152339 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:31.842229 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:31.842199 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:32.441918 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:32.441885 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:33.046470 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:33.046429 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:33.657031 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:33.656994 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:34.246599 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:34.246569 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:34.847118 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:34.847087 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:34.950896 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:34.950859 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:35.558974 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:35.558943 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:35.664339 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:35.664305 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:35.765802 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:35.765601 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:36.012605 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:36.012572 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:37.019022 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.018985 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm"] Oct 31 00:58:37.019469 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.019367 4519 cpu_manager.go:414] "containerMap: removing container" podUID="394705fd-8f0a-421c-83d6-e43d1fa69e4b" containerName="main" Oct 31 00:58:37.019469 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.019384 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394705fd-8f0a-421c-83d6-e43d1fa69e4b" containerName="main" Oct 31 00:58:37.019469 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.019413 4519 cpu_manager.go:414] "containerMap: removing container" podUID="d46149f1-da3b-4442-bdd4-129a16c73b57" containerName="payload-pre-processing" Oct 31 00:58:37.019469 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.019418 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46149f1-da3b-4442-bdd4-129a16c73b57" containerName="payload-pre-processing" Oct 31 00:58:37.019469 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.019472 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="394705fd-8f0a-421c-83d6-e43d1fa69e4b" containerName="main" Oct 31 00:58:37.019654 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.019483 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d46149f1-da3b-4442-bdd4-129a16c73b57" containerName="payload-pre-processing" Oct 31 00:58:37.024275 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.024258 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.038869 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.038843 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-1919c8\"/\"openshift-service-ca.crt\"" Oct 31 00:58:37.039003 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.038956 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-1919c8\"/\"kube-root-ca.crt\"" Oct 31 00:58:37.039067 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.039026 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-1919c8\"/\"simulator-single-main-1919c8-kserve-self-signed-certs\"" Oct 31 00:58:37.039120 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.039068 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-1919c8\"/\"default-dockercfg-dd49p\"" Oct 31 00:58:37.039265 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.039246 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm"] Oct 31 00:58:37.078137 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.078102 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:37.136093 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.136061 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-dshm\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.136093 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.136103 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb8f8\" (UniqueName: \"kubernetes.io/projected/ba2839f3-9153-4d39-820e-06bdb31a82d9-kube-api-access-qb8f8\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.136354 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.136129 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba2839f3-9153-4d39-820e-06bdb31a82d9-tls-certs\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.136354 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.136207 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-model-cache\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.136354 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.136322 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-tmp-dir\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.136354 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.136347 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-home\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.238219 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.238173 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-tmp-dir\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.238422 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.238232 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-home\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.238422 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.238273 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-dshm\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.238422 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.238307 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb8f8\" (UniqueName: \"kubernetes.io/projected/ba2839f3-9153-4d39-820e-06bdb31a82d9-kube-api-access-qb8f8\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.238422 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.238344 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba2839f3-9153-4d39-820e-06bdb31a82d9-tls-certs\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.238741 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.238713 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-model-cache\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.238848 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.238798 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-home\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.238848 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.238797 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-tmp-dir\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.239230 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.239108 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-model-cache\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.243204 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.243178 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-dshm\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.256530 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.256503 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb8f8\" (UniqueName: \"kubernetes.io/projected/ba2839f3-9153-4d39-820e-06bdb31a82d9-kube-api-access-qb8f8\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.256530 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.256519 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba2839f3-9153-4d39-820e-06bdb31a82d9-tls-certs\") pod \"simulator-single-main-1919c8-kserve-744b864db5-zsszm\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.347213 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.347138 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:37.464860 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.464780 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:37.628594 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:37.628567 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm"] Oct 31 00:58:37.630974 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:58:37.630944 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba2839f3_9153_4d39_820e_06bdb31a82d9.slice/crio-3bc98d7ad7e255d71a68f9a3edd1241a2f43156b97809f843b7673002c685303 WatchSource:0}: Error finding container 3bc98d7ad7e255d71a68f9a3edd1241a2f43156b97809f843b7673002c685303: Status 404 returned error can't find the container with id 3bc98d7ad7e255d71a68f9a3edd1241a2f43156b97809f843b7673002c685303 Oct 31 00:58:38.167861 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:38.167827 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" event={"ID":"ba2839f3-9153-4d39-820e-06bdb31a82d9","Type":"ContainerStarted","Data":"48ce0730a11e92da87a9128800a4edd1621ede4fe58e8d1dd997c295bd2a2644"} Oct 31 00:58:38.168299 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:38.167871 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" event={"ID":"ba2839f3-9153-4d39-820e-06bdb31a82d9","Type":"ContainerStarted","Data":"3bc98d7ad7e255d71a68f9a3edd1241a2f43156b97809f843b7673002c685303"} Oct 31 00:58:38.195176 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:38.195113 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" podStartSLOduration=2.195095825 podStartE2EDuration="2.195095825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:38.193956004 +0000 UTC m=+1514.223136394" watchObservedRunningTime="2026-10-31 00:58:38.195095825 +0000 UTC m=+1514.224276216" Oct 31 00:58:38.265627 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:38.265549 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:38.347483 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:38.347433 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:39.083604 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:39.083571 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:39.871228 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:39.871190 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:40.796066 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:40.796016 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:41.546720 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:41.546683 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:42.168515 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:42.168483 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:42.836792 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:42.836756 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:43.559924 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:43.559892 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:44.233356 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:44.233320 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:44.844589 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:44.844551 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:45.452294 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:45.452259 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:46.144526 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:46.144491 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:46.760551 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:46.760518 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:47.347653 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:47.347610 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:47.359921 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:47.359890 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:47.540323 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:58:47.540286 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:58:47.555325 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:47.555297 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:48.238941 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:48.238902 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:48.243953 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:48.243929 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:58:48.861311 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:48.861278 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:49.602437 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:49.602378 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:50.241511 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:50.241477 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:50.849031 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:50.848998 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:51.562015 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:51.561968 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:52.232117 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:52.232086 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:52.841186 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:52.841157 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:53.626883 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:53.626844 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:54.357537 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:54.357503 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:54.970846 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:54.970808 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:55.952676 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:55.952620 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:56.676882 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:56.676846 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:57.471015 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:57.470982 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:58.148149 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:58.148114 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:58.855263 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:58.855229 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:59.674212 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:58:59.674180 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:00.357273 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:00.357242 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:01.074675 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:01.074630 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:01.937493 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:01.937455 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:02.656245 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:02.656208 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:03.029122 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:03.029081 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn"] Oct 31 00:59:03.031192 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:03.031164 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-sljzn"] Oct 31 00:59:03.355933 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:03.355840 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:04.100556 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:04.100526 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2028f40f-a603-4ba9-bf1b-a3c0e95f1aab" path="/var/lib/kubelet/pods/2028f40f-a603-4ba9-bf1b-a3c0e95f1aab/volumes" Oct 31 00:59:04.160518 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:04.160484 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:04.859313 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:04.859281 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:05.678903 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:05.678867 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:06.588053 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:06.588015 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:07.818470 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:07.818440 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:08.061564 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.061511 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2"] Oct 31 00:59:08.066490 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.066457 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" Oct 31 00:59:08.083946 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.083912 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2"] Oct 31 00:59:08.091149 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.091121 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:59:08.091489 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.091467 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:59:08.126787 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.126749 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6251df92-be00-4d76-8346-3ce0c4c9f528-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e0c5581-vvpn2\" (UID: \"6251df92-be00-4d76-8346-3ce0c4c9f528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" Oct 31 00:59:08.126984 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.126815 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6251df92-be00-4d76-8346-3ce0c4c9f528-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e0c5581-vvpn2\" (UID: \"6251df92-be00-4d76-8346-3ce0c4c9f528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" Oct 31 00:59:08.126984 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.126919 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf7p2\" (UniqueName: \"kubernetes.io/projected/6251df92-be00-4d76-8346-3ce0c4c9f528-kube-api-access-kf7p2\") pod \"maas-api-revoke-sub-0e0c5581-vvpn2\" (UID: \"6251df92-be00-4d76-8346-3ce0c4c9f528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" Oct 31 00:59:08.139489 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.139458 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:59:08.139489 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.139480 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:59:08.227842 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.227800 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf7p2\" (UniqueName: \"kubernetes.io/projected/6251df92-be00-4d76-8346-3ce0c4c9f528-kube-api-access-kf7p2\") pod \"maas-api-revoke-sub-0e0c5581-vvpn2\" (UID: \"6251df92-be00-4d76-8346-3ce0c4c9f528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" Oct 31 00:59:08.228045 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.227886 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6251df92-be00-4d76-8346-3ce0c4c9f528-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e0c5581-vvpn2\" (UID: \"6251df92-be00-4d76-8346-3ce0c4c9f528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" Oct 31 00:59:08.228045 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.227944 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6251df92-be00-4d76-8346-3ce0c4c9f528-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e0c5581-vvpn2\" (UID: \"6251df92-be00-4d76-8346-3ce0c4c9f528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" Oct 31 00:59:08.228773 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.228748 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6251df92-be00-4d76-8346-3ce0c4c9f528-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e0c5581-vvpn2\" (UID: \"6251df92-be00-4d76-8346-3ce0c4c9f528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" Oct 31 00:59:08.230650 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.230614 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6251df92-be00-4d76-8346-3ce0c4c9f528-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e0c5581-vvpn2\" (UID: \"6251df92-be00-4d76-8346-3ce0c4c9f528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" Oct 31 00:59:08.243121 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.243085 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf7p2\" (UniqueName: \"kubernetes.io/projected/6251df92-be00-4d76-8346-3ce0c4c9f528-kube-api-access-kf7p2\") pod \"maas-api-revoke-sub-0e0c5581-vvpn2\" (UID: \"6251df92-be00-4d76-8346-3ce0c4c9f528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" Oct 31 00:59:08.379678 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.379565 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" Oct 31 00:59:08.654663 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.654616 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2"] Oct 31 00:59:08.657361 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:59:08.657327 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6251df92_be00_4d76_8346_3ce0c4c9f528.slice/crio-ae28f0cc15b20d8a433c8ee9790ec4e5a16158f0a91d813c6460583ec10c1316 WatchSource:0}: Error finding container ae28f0cc15b20d8a433c8ee9790ec4e5a16158f0a91d813c6460583ec10c1316: Status 404 returned error can't find the container with id ae28f0cc15b20d8a433c8ee9790ec4e5a16158f0a91d813c6460583ec10c1316 Oct 31 00:59:08.798666 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:08.798606 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:09.326844 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:09.326804 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" event={"ID":"6251df92-be00-4d76-8346-3ce0c4c9f528","Type":"ContainerStarted","Data":"748907ab8512d4f0246a87a0b583e678cc191c39072eefba071c70106d41d449"} Oct 31 00:59:09.327247 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:09.326865 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" event={"ID":"6251df92-be00-4d76-8346-3ce0c4c9f528","Type":"ContainerStarted","Data":"ae28f0cc15b20d8a433c8ee9790ec4e5a16158f0a91d813c6460583ec10c1316"} Oct 31 00:59:09.351128 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:09.351065 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" podStartSLOduration=1.351051518 podStartE2EDuration="1.351051518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:09.349421164 +0000 UTC m=+1545.378601555" watchObservedRunningTime="2026-10-31 00:59:09.351051518 +0000 UTC m=+1545.380231907" Oct 31 00:59:09.768289 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:09.768257 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:10.332381 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:10.332346 4519 generic.go:356] "Generic (PLEG): container finished" podID="6251df92-be00-4d76-8346-3ce0c4c9f528" containerID="748907ab8512d4f0246a87a0b583e678cc191c39072eefba071c70106d41d449" exitCode=0 Oct 31 00:59:10.332838 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:10.332437 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" event={"ID":"6251df92-be00-4d76-8346-3ce0c4c9f528","Type":"ContainerDied","Data":"748907ab8512d4f0246a87a0b583e678cc191c39072eefba071c70106d41d449"} Oct 31 00:59:10.790121 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:10.790088 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:11.465228 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:11.465200 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" Oct 31 00:59:11.514939 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:11.514906 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:11.560186 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:11.560151 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6251df92-be00-4d76-8346-3ce0c4c9f528-kube-api-access-kf7p2\" (UniqueName: \"kubernetes.io/projected/6251df92-be00-4d76-8346-3ce0c4c9f528-kube-api-access-kf7p2\") pod \"6251df92-be00-4d76-8346-3ce0c4c9f528\" (UID: \"6251df92-be00-4d76-8346-3ce0c4c9f528\") " Oct 31 00:59:11.560454 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:11.560265 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6251df92-be00-4d76-8346-3ce0c4c9f528-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6251df92-be00-4d76-8346-3ce0c4c9f528-maas-api-xks-ca\") pod \"6251df92-be00-4d76-8346-3ce0c4c9f528\" (UID: \"6251df92-be00-4d76-8346-3ce0c4c9f528\") " Oct 31 00:59:11.560454 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:11.560303 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6251df92-be00-4d76-8346-3ce0c4c9f528-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6251df92-be00-4d76-8346-3ce0c4c9f528-maas-api-openshift-ca\") pod \"6251df92-be00-4d76-8346-3ce0c4c9f528\" (UID: \"6251df92-be00-4d76-8346-3ce0c4c9f528\") " Oct 31 00:59:11.560789 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:11.560755 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6251df92-be00-4d76-8346-3ce0c4c9f528-maas-api-openshift-ca" pod "6251df92-be00-4d76-8346-3ce0c4c9f528" (UID: "6251df92-be00-4d76-8346-3ce0c4c9f528"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:11.563058 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:11.563022 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6251df92-be00-4d76-8346-3ce0c4c9f528-maas-api-xks-ca" pod "6251df92-be00-4d76-8346-3ce0c4c9f528" (UID: "6251df92-be00-4d76-8346-3ce0c4c9f528"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:59:11.563830 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:11.563807 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6251df92-be00-4d76-8346-3ce0c4c9f528-kube-api-access-kf7p2" pod "6251df92-be00-4d76-8346-3ce0c4c9f528" (UID: "6251df92-be00-4d76-8346-3ce0c4c9f528"). InnerVolumeSpecName "kube-api-access-kf7p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:11.661102 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:11.661020 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6251df92-be00-4d76-8346-3ce0c4c9f528-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:11.661102 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:11.661049 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6251df92-be00-4d76-8346-3ce0c4c9f528-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:11.661102 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:11.661060 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf7p2\" (UniqueName: \"kubernetes.io/projected/6251df92-be00-4d76-8346-3ce0c4c9f528-kube-api-access-kf7p2\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:12.144152 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:12.144114 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:12.347578 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:12.347544 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" Oct 31 00:59:12.347578 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:12.347559 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2" event={"ID":"6251df92-be00-4d76-8346-3ce0c4c9f528","Type":"ContainerDied","Data":"ae28f0cc15b20d8a433c8ee9790ec4e5a16158f0a91d813c6460583ec10c1316"} Oct 31 00:59:12.347824 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:12.347656 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae28f0cc15b20d8a433c8ee9790ec4e5a16158f0a91d813c6460583ec10c1316" Oct 31 00:59:12.354904 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:12.354862 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:13.102074 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:13.102042 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:13.881287 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:13.881245 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:14.059418 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:14.059382 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm"] Oct 31 00:59:14.062809 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:14.062782 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vvcmm"] Oct 31 00:59:14.101230 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:14.101197 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b25b67ad-adf1-447b-91e2-7d33e465a641" path="/var/lib/kubelet/pods/b25b67ad-adf1-447b-91e2-7d33e465a641/volumes" Oct 31 00:59:14.798185 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:14.798153 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:15.676980 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:15.676945 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:16.474260 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:16.474224 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:17.334108 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:17.334072 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:17.540852 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:59:17.540820 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:59:18.090956 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:18.090914 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:18.867230 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:18.867169 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:19.119007 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.118922 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl"] Oct 31 00:59:19.119283 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.119270 4519 cpu_manager.go:414] "containerMap: removing container" podUID="6251df92-be00-4d76-8346-3ce0c4c9f528" containerName="revoke-keys" Oct 31 00:59:19.119329 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.119286 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6251df92-be00-4d76-8346-3ce0c4c9f528" containerName="revoke-keys" Oct 31 00:59:19.119362 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.119348 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6251df92-be00-4d76-8346-3ce0c4c9f528" containerName="revoke-keys" Oct 31 00:59:19.122316 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.122299 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" Oct 31 00:59:19.125915 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.125887 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:19.128646 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.128611 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:59:19.147234 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.147193 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 00:59:19.147474 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.147284 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:59:19.152077 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.152045 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl"] Oct 31 00:59:19.229574 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.229536 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:59:19.236556 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.236512 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecacaf61-0240-4d92-8aec-ad3753842d75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl\" (UID: \"ecacaf61-0240-4d92-8aec-ad3753842d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" Oct 31 00:59:19.236715 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.236560 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skzbq\" (UniqueName: \"kubernetes.io/projected/ecacaf61-0240-4d92-8aec-ad3753842d75-kube-api-access-skzbq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl\" (UID: \"ecacaf61-0240-4d92-8aec-ad3753842d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" Oct 31 00:59:19.236715 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.236694 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecacaf61-0240-4d92-8aec-ad3753842d75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl\" (UID: \"ecacaf61-0240-4d92-8aec-ad3753842d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" Oct 31 00:59:19.337935 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.337898 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecacaf61-0240-4d92-8aec-ad3753842d75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl\" (UID: \"ecacaf61-0240-4d92-8aec-ad3753842d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" Oct 31 00:59:19.337935 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.337940 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skzbq\" (UniqueName: \"kubernetes.io/projected/ecacaf61-0240-4d92-8aec-ad3753842d75-kube-api-access-skzbq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl\" (UID: \"ecacaf61-0240-4d92-8aec-ad3753842d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" Oct 31 00:59:19.338224 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.338075 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecacaf61-0240-4d92-8aec-ad3753842d75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl\" (UID: \"ecacaf61-0240-4d92-8aec-ad3753842d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" Oct 31 00:59:19.338696 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.338670 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecacaf61-0240-4d92-8aec-ad3753842d75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl\" (UID: \"ecacaf61-0240-4d92-8aec-ad3753842d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" Oct 31 00:59:19.340465 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.340445 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecacaf61-0240-4d92-8aec-ad3753842d75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl\" (UID: \"ecacaf61-0240-4d92-8aec-ad3753842d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" Oct 31 00:59:19.356713 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.356681 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skzbq\" (UniqueName: \"kubernetes.io/projected/ecacaf61-0240-4d92-8aec-ad3753842d75-kube-api-access-skzbq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl\" (UID: \"ecacaf61-0240-4d92-8aec-ad3753842d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" Oct 31 00:59:19.434955 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.434864 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" Oct 31 00:59:19.751196 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:19.740427 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl"] Oct 31 00:59:19.751196 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:59:19.741023 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecacaf61_0240_4d92_8aec_ad3753842d75.slice/crio-e5bc8bd115749a059aa0049ae1a3c5562d5403c8125579673f8183a483583d6c WatchSource:0}: Error finding container e5bc8bd115749a059aa0049ae1a3c5562d5403c8125579673f8183a483583d6c: Status 404 returned error can't find the container with id e5bc8bd115749a059aa0049ae1a3c5562d5403c8125579673f8183a483583d6c Oct 31 00:59:20.191253 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:20.191147 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:20.388012 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:20.387968 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" event={"ID":"ecacaf61-0240-4d92-8aec-ad3753842d75","Type":"ContainerStarted","Data":"0f2f2e9ad3065840127102fa4f5cf987c941396279e2f218e20d30280efca0df"} Oct 31 00:59:20.388170 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:20.388034 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" event={"ID":"ecacaf61-0240-4d92-8aec-ad3753842d75","Type":"ContainerStarted","Data":"e5bc8bd115749a059aa0049ae1a3c5562d5403c8125579673f8183a483583d6c"} Oct 31 00:59:20.408374 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:20.408318 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" podStartSLOduration=1.40830283 podStartE2EDuration="1.40830283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:20.406447489 +0000 UTC m=+1556.435627878" watchObservedRunningTime="2026-10-31 00:59:20.40830283 +0000 UTC m=+1556.437483221" Oct 31 00:59:21.082911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:21.082874 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:21.400072 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:21.399971 4519 generic.go:356] "Generic (PLEG): container finished" podID="ecacaf61-0240-4d92-8aec-ad3753842d75" containerID="0f2f2e9ad3065840127102fa4f5cf987c941396279e2f218e20d30280efca0df" exitCode=0 Oct 31 00:59:21.400072 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:21.400058 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" event={"ID":"ecacaf61-0240-4d92-8aec-ad3753842d75","Type":"ContainerDied","Data":"0f2f2e9ad3065840127102fa4f5cf987c941396279e2f218e20d30280efca0df"} Oct 31 00:59:21.979203 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:21.979168 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:22.533832 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:22.533806 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" Oct 31 00:59:22.672749 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:22.672627 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecacaf61-0240-4d92-8aec-ad3753842d75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecacaf61-0240-4d92-8aec-ad3753842d75-maas-api-openshift-ca\") pod \"ecacaf61-0240-4d92-8aec-ad3753842d75\" (UID: \"ecacaf61-0240-4d92-8aec-ad3753842d75\") " Oct 31 00:59:22.672749 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:22.672708 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecacaf61-0240-4d92-8aec-ad3753842d75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecacaf61-0240-4d92-8aec-ad3753842d75-maas-api-xks-ca\") pod \"ecacaf61-0240-4d92-8aec-ad3753842d75\" (UID: \"ecacaf61-0240-4d92-8aec-ad3753842d75\") " Oct 31 00:59:22.672749 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:22.672750 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecacaf61-0240-4d92-8aec-ad3753842d75-kube-api-access-skzbq\" (UniqueName: \"kubernetes.io/projected/ecacaf61-0240-4d92-8aec-ad3753842d75-kube-api-access-skzbq\") pod \"ecacaf61-0240-4d92-8aec-ad3753842d75\" (UID: \"ecacaf61-0240-4d92-8aec-ad3753842d75\") " Oct 31 00:59:22.673093 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:22.673066 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecacaf61-0240-4d92-8aec-ad3753842d75-maas-api-openshift-ca" pod "ecacaf61-0240-4d92-8aec-ad3753842d75" (UID: "ecacaf61-0240-4d92-8aec-ad3753842d75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:22.674840 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:22.674809 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecacaf61-0240-4d92-8aec-ad3753842d75-maas-api-xks-ca" pod "ecacaf61-0240-4d92-8aec-ad3753842d75" (UID: "ecacaf61-0240-4d92-8aec-ad3753842d75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:59:22.674941 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:22.674834 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecacaf61-0240-4d92-8aec-ad3753842d75-kube-api-access-skzbq" pod "ecacaf61-0240-4d92-8aec-ad3753842d75" (UID: "ecacaf61-0240-4d92-8aec-ad3753842d75"). InnerVolumeSpecName "kube-api-access-skzbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:22.773651 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:22.773608 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecacaf61-0240-4d92-8aec-ad3753842d75-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:22.773820 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:22.773671 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skzbq\" (UniqueName: \"kubernetes.io/projected/ecacaf61-0240-4d92-8aec-ad3753842d75-kube-api-access-skzbq\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:22.773820 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:22.773689 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecacaf61-0240-4d92-8aec-ad3753842d75-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:22.933872 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:22.933146 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:23.410549 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:23.410515 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" event={"ID":"ecacaf61-0240-4d92-8aec-ad3753842d75","Type":"ContainerDied","Data":"e5bc8bd115749a059aa0049ae1a3c5562d5403c8125579673f8183a483583d6c"} Oct 31 00:59:23.410549 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:23.410540 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl" Oct 31 00:59:23.410817 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:23.410572 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5bc8bd115749a059aa0049ae1a3c5562d5403c8125579673f8183a483583d6c" Oct 31 00:59:23.763773 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:23.763737 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:23.872131 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:23.872094 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm"] Oct 31 00:59:23.965187 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:23.965158 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:24.634395 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:24.634351 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:25.372013 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:25.371972 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:26.243727 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:26.243694 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:26.294504 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:26.294468 4519 scope.go:122] "RemoveContainer" containerID="b4aef96d223500c54fb826992760cb806bab2156a8459ef216ee1becae496590" Oct 31 00:59:26.302758 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:26.302729 4519 scope.go:122] "RemoveContainer" containerID="8253bb22b3a1f6c1a8df546ac2a550f6dbc9ea03806227b656e42e9d5a140933" Oct 31 00:59:26.309964 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:26.309935 4519 scope.go:122] "RemoveContainer" containerID="21aa6caea6fed8afe33f0e90d38736608563244e870ab62aa420c14a88bae1aa" Oct 31 00:59:26.972897 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:26.972865 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:27.816029 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:27.815997 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:28.548208 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:28.548180 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:29.249703 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:29.249671 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:29.373665 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:29.373615 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:29.474031 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:29.473985 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:29.850036 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:29.849996 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:30.148911 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:30.148836 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:30.203867 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:30.203836 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl"] Oct 31 00:59:30.206469 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:30.206438 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1919c8-4nwtl"] Oct 31 00:59:30.244003 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:30.243975 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:30.780153 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:30.780118 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:32.100463 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:32.100430 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecacaf61-0240-4d92-8aec-ad3753842d75" path="/var/lib/kubelet/pods/ecacaf61-0240-4d92-8aec-ad3753842d75/volumes" Oct 31 00:59:37.272823 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:37.272740 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:37.404691 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:37.404654 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:37.644446 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:37.644358 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:38.891593 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:38.891553 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" podUID="ba2839f3-9153-4d39-820e-06bdb31a82d9" containerName="main" containerID="cri-o://48ce0730a11e92da87a9128800a4edd1621ede4fe58e8d1dd997c295bd2a2644" gracePeriod=45 Oct 31 00:59:39.140540 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.140512 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:59:39.218231 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.218134 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-home\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-home\") pod \"ba2839f3-9153-4d39-820e-06bdb31a82d9\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " Oct 31 00:59:39.218231 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.218191 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-tmp-dir\") pod \"ba2839f3-9153-4d39-820e-06bdb31a82d9\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " Oct 31 00:59:39.218508 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.218275 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba2839f3-9153-4d39-820e-06bdb31a82d9-kube-api-access-qb8f8\" (UniqueName: \"kubernetes.io/projected/ba2839f3-9153-4d39-820e-06bdb31a82d9-kube-api-access-qb8f8\") pod \"ba2839f3-9153-4d39-820e-06bdb31a82d9\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " Oct 31 00:59:39.218508 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.218297 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-model-cache\") pod \"ba2839f3-9153-4d39-820e-06bdb31a82d9\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " Oct 31 00:59:39.218508 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.218349 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-dshm\") pod \"ba2839f3-9153-4d39-820e-06bdb31a82d9\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " Oct 31 00:59:39.218508 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.218380 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba2839f3-9153-4d39-820e-06bdb31a82d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba2839f3-9153-4d39-820e-06bdb31a82d9-tls-certs\") pod \"ba2839f3-9153-4d39-820e-06bdb31a82d9\" (UID: \"ba2839f3-9153-4d39-820e-06bdb31a82d9\") " Oct 31 00:59:39.218508 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.218447 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-home" pod "ba2839f3-9153-4d39-820e-06bdb31a82d9" (UID: "ba2839f3-9153-4d39-820e-06bdb31a82d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:39.218804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.218587 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-tmp-dir" pod "ba2839f3-9153-4d39-820e-06bdb31a82d9" (UID: "ba2839f3-9153-4d39-820e-06bdb31a82d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:39.218804 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.218619 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-model-cache" pod "ba2839f3-9153-4d39-820e-06bdb31a82d9" (UID: "ba2839f3-9153-4d39-820e-06bdb31a82d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:39.218909 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.218856 4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-tmp-dir\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:39.218909 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.218874 4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-model-cache\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:39.218909 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.218888 4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-home\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:39.220650 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.220613 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-dshm" pod "ba2839f3-9153-4d39-820e-06bdb31a82d9" (UID: "ba2839f3-9153-4d39-820e-06bdb31a82d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:39.220797 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.220718 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba2839f3-9153-4d39-820e-06bdb31a82d9-kube-api-access-qb8f8" pod "ba2839f3-9153-4d39-820e-06bdb31a82d9" (UID: "ba2839f3-9153-4d39-820e-06bdb31a82d9"). InnerVolumeSpecName "kube-api-access-qb8f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:39.221277 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.221253 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba2839f3-9153-4d39-820e-06bdb31a82d9-tls-certs" pod "ba2839f3-9153-4d39-820e-06bdb31a82d9" (UID: "ba2839f3-9153-4d39-820e-06bdb31a82d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:59:39.319630 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.319586 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb8f8\" (UniqueName: \"kubernetes.io/projected/ba2839f3-9153-4d39-820e-06bdb31a82d9-kube-api-access-qb8f8\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:39.319630 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.319622 4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba2839f3-9153-4d39-820e-06bdb31a82d9-dshm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:39.319630 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.319666 4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba2839f3-9153-4d39-820e-06bdb31a82d9-tls-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:39.479520 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.479424 4519 generic.go:356] "Generic (PLEG): container finished" podID="ba2839f3-9153-4d39-820e-06bdb31a82d9" containerID="48ce0730a11e92da87a9128800a4edd1621ede4fe58e8d1dd997c295bd2a2644" exitCode=0 Oct 31 00:59:39.479520 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.479481 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" Oct 31 00:59:39.479739 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.479515 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" event={"ID":"ba2839f3-9153-4d39-820e-06bdb31a82d9","Type":"ContainerDied","Data":"48ce0730a11e92da87a9128800a4edd1621ede4fe58e8d1dd997c295bd2a2644"} Oct 31 00:59:39.479739 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.479563 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm" event={"ID":"ba2839f3-9153-4d39-820e-06bdb31a82d9","Type":"ContainerDied","Data":"3bc98d7ad7e255d71a68f9a3edd1241a2f43156b97809f843b7673002c685303"} Oct 31 00:59:39.479739 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.479580 4519 scope.go:122] "RemoveContainer" containerID="48ce0730a11e92da87a9128800a4edd1621ede4fe58e8d1dd997c295bd2a2644" Oct 31 00:59:39.488086 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.488063 4519 scope.go:122] "RemoveContainer" containerID="48ce0730a11e92da87a9128800a4edd1621ede4fe58e8d1dd997c295bd2a2644" Oct 31 00:59:39.488368 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:59:39.488348 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ce0730a11e92da87a9128800a4edd1621ede4fe58e8d1dd997c295bd2a2644\": container with ID starting with 48ce0730a11e92da87a9128800a4edd1621ede4fe58e8d1dd997c295bd2a2644 not found: ID does not exist" containerID="48ce0730a11e92da87a9128800a4edd1621ede4fe58e8d1dd997c295bd2a2644" Oct 31 00:59:39.488432 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.488382 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ce0730a11e92da87a9128800a4edd1621ede4fe58e8d1dd997c295bd2a2644"} err="failed to get container status \"48ce0730a11e92da87a9128800a4edd1621ede4fe58e8d1dd997c295bd2a2644\": rpc error: code = NotFound desc = could not find container \"48ce0730a11e92da87a9128800a4edd1621ede4fe58e8d1dd997c295bd2a2644\": container with ID starting with 48ce0730a11e92da87a9128800a4edd1621ede4fe58e8d1dd997c295bd2a2644 not found: ID does not exist" Oct 31 00:59:39.501026 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.500990 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm"] Oct 31 00:59:39.504159 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:39.504133 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-1919c8/simulator-single-main-1919c8-kserve-744b864db5-zsszm"] Oct 31 00:59:40.100534 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:40.100498 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba2839f3-9153-4d39-820e-06bdb31a82d9" path="/var/lib/kubelet/pods/ba2839f3-9153-4d39-820e-06bdb31a82d9/volumes" Oct 31 00:59:41.506031 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.505995 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:41.812745 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.812666 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh"] Oct 31 00:59:41.812996 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.812983 4519 cpu_manager.go:414] "containerMap: removing container" podUID="ba2839f3-9153-4d39-820e-06bdb31a82d9" containerName="main" Oct 31 00:59:41.813071 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.812998 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba2839f3-9153-4d39-820e-06bdb31a82d9" containerName="main" Oct 31 00:59:41.813071 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.813012 4519 cpu_manager.go:414] "containerMap: removing container" podUID="ecacaf61-0240-4d92-8aec-ad3753842d75" containerName="revoke-keys" Oct 31 00:59:41.813071 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.813017 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecacaf61-0240-4d92-8aec-ad3753842d75" containerName="revoke-keys" Oct 31 00:59:41.813195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.813092 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba2839f3-9153-4d39-820e-06bdb31a82d9" containerName="main" Oct 31 00:59:41.813195 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.813101 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecacaf61-0240-4d92-8aec-ad3753842d75" containerName="revoke-keys" Oct 31 00:59:41.817270 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.817248 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.825342 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.825312 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-66c4l\"" Oct 31 00:59:41.828234 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.828211 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh"] Oct 31 00:59:41.834282 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.833529 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 00:59:41.834282 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.833755 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2f32c431-kserve-self-signed-certs\"" Oct 31 00:59:41.834282 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.834204 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 00:59:41.842941 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.842907 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d7c60c-14c0-4287-8c28-88f17fbe205e-tls-certs\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.842941 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.842943 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-dshm\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.843201 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.842970 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t59xq\" (UniqueName: \"kubernetes.io/projected/96d7c60c-14c0-4287-8c28-88f17fbe205e-kube-api-access-t59xq\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.843201 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.843071 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-home\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.843201 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.843095 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-model-cache\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.843201 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.843132 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-tmp-dir\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.876062 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.876005 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:41.944205 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.944153 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d7c60c-14c0-4287-8c28-88f17fbe205e-tls-certs\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.944205 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.944204 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-dshm\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.944888 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.944232 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t59xq\" (UniqueName: \"kubernetes.io/projected/96d7c60c-14c0-4287-8c28-88f17fbe205e-kube-api-access-t59xq\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.944888 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.944288 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-home\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.944888 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.944313 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-model-cache\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.944888 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.944366 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-tmp-dir\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.958379 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.956048 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-tmp-dir\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.958379 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.958312 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-home\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.958910 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.958846 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-dshm\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.959233 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.959208 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-model-cache\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.961997 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.961965 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d7c60c-14c0-4287-8c28-88f17fbe205e-tls-certs\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:41.972583 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:41.972230 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t59xq\" (UniqueName: \"kubernetes.io/projected/96d7c60c-14c0-4287-8c28-88f17fbe205e-kube-api-access-t59xq\") pod \"e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:42.137421 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:42.137326 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:42.412293 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:42.412264 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh"] Oct 31 00:59:42.414590 ip-10-0-143-188 kubenswrapper[4519]: W1031 00:59:42.414562 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96d7c60c_14c0_4287_8c28_88f17fbe205e.slice/crio-583394cd99acaea1dbcf321668975ba85f06061f68d46061eba90637e8a4324c WatchSource:0}: Error finding container 583394cd99acaea1dbcf321668975ba85f06061f68d46061eba90637e8a4324c: Status 404 returned error can't find the container with id 583394cd99acaea1dbcf321668975ba85f06061f68d46061eba90637e8a4324c Oct 31 00:59:42.495718 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:42.495673 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" event={"ID":"96d7c60c-14c0-4287-8c28-88f17fbe205e","Type":"ContainerStarted","Data":"4922cfb0f36b2f7c073c312e7164dab7bb52f44dc0371c7e868eefb3dab05e6c"} Oct 31 00:59:42.495884 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:42.495735 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" event={"ID":"96d7c60c-14c0-4287-8c28-88f17fbe205e","Type":"ContainerStarted","Data":"583394cd99acaea1dbcf321668975ba85f06061f68d46061eba90637e8a4324c"} Oct 31 00:59:42.515671 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:42.515603 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" podStartSLOduration=1.515587698 podStartE2EDuration="1.515587698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:42.514215434 +0000 UTC m=+1578.543395826" watchObservedRunningTime="2026-10-31 00:59:42.515587698 +0000 UTC m=+1578.544768088" Oct 31 00:59:43.138473 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:43.138438 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:47.542752 ip-10-0-143-188 kubenswrapper[4519]: E1031 00:59:47.542711 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 00:59:52.138454 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:52.138420 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:52.150778 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:52.150754 4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:52.553962 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:52.553932 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 00:59:59.535488 ip-10-0-143-188 kubenswrapper[4519]: I1031 00:59:59.535437 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:00.681304 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:00.681273 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:00.780432 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:00.780396 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:02.141034 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:02.141003 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:08.028917 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:08.028883 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:08.177690 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:08.177645 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:08.279232 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:08.279150 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:12.168054 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:12.168022 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:15.675285 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:15.675248 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt"] Oct 31 01:00:15.678453 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:15.678434 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" Oct 31 01:00:15.686754 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:15.686723 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:00:15.686958 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:15.686776 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slc7v\"" Oct 31 01:00:15.687051 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:15.686871 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:00:15.687208 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:15.687194 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:00:15.690052 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:15.690029 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt"] Oct 31 01:00:15.810976 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:15.810932 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqqkw\" (UniqueName: \"kubernetes.io/projected/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-kube-api-access-qqqkw\") pod \"maas-api-revoke-sub-452b6d7b-gnzpt\" (UID: \"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" Oct 31 01:00:15.810976 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:15.810982 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-gnzpt\" (UID: \"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" Oct 31 01:00:15.811247 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:15.811041 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-gnzpt\" (UID: \"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" Oct 31 01:00:15.912565 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:15.912521 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqkw\" (UniqueName: \"kubernetes.io/projected/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-kube-api-access-qqqkw\") pod \"maas-api-revoke-sub-452b6d7b-gnzpt\" (UID: \"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" Oct 31 01:00:15.912565 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:15.912569 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-gnzpt\" (UID: \"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" Oct 31 01:00:15.912815 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:15.912604 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-gnzpt\" (UID: \"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" Oct 31 01:00:15.913443 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:15.913412 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-gnzpt\" (UID: \"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" Oct 31 01:00:15.915100 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:15.915074 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-gnzpt\" (UID: \"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" Oct 31 01:00:15.921950 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:15.921929 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqqkw\" (UniqueName: \"kubernetes.io/projected/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-kube-api-access-qqqkw\") pod \"maas-api-revoke-sub-452b6d7b-gnzpt\" (UID: \"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" Oct 31 01:00:16.000491 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:16.000454 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" Oct 31 01:00:16.274876 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:16.274843 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt"] Oct 31 01:00:16.276807 ip-10-0-143-188 kubenswrapper[4519]: W1031 01:00:16.276782 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0cf260_9d68_4a89_9e8c_5c8f43496ec1.slice/crio-9418e7ceb7cd0e0e099cac696834ed41259afbebbc28b34978286319cd4d5660 WatchSource:0}: Error finding container 9418e7ceb7cd0e0e099cac696834ed41259afbebbc28b34978286319cd4d5660: Status 404 returned error can't find the container with id 9418e7ceb7cd0e0e099cac696834ed41259afbebbc28b34978286319cd4d5660 Oct 31 01:00:16.665494 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:16.665449 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" event={"ID":"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1","Type":"ContainerStarted","Data":"b7ea4d52a9f79a95918c112cc47e349f16bd2bd8025727b4b0b979fe6dbc607e"} Oct 31 01:00:16.665719 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:16.665515 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" event={"ID":"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1","Type":"ContainerStarted","Data":"9418e7ceb7cd0e0e099cac696834ed41259afbebbc28b34978286319cd4d5660"} Oct 31 01:00:16.686423 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:16.686360 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" podStartSLOduration=1.6863431740000001 podStartE2EDuration="1.686343174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:16.68440231 +0000 UTC m=+1612.713582715" watchObservedRunningTime="2026-10-31 01:00:16.686343174 +0000 UTC m=+1612.715523568" Oct 31 01:00:17.560893 ip-10-0-143-188 kubenswrapper[4519]: E1031 01:00:17.560843 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 01:00:17.697461 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:17.697034 4519 generic.go:356] "Generic (PLEG): container finished" podID="5e0cf260-9d68-4a89-9e8c-5c8f43496ec1" containerID="b7ea4d52a9f79a95918c112cc47e349f16bd2bd8025727b4b0b979fe6dbc607e" exitCode=0 Oct 31 01:00:17.697461 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:17.697085 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" event={"ID":"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1","Type":"ContainerDied","Data":"b7ea4d52a9f79a95918c112cc47e349f16bd2bd8025727b4b0b979fe6dbc607e"} Oct 31 01:00:19.019072 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:19.019049 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" Oct 31 01:00:19.107661 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:19.106771 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-maas-api-openshift-ca\") pod \"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1\" (UID: \"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1\") " Oct 31 01:00:19.107661 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:19.107291 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-maas-api-openshift-ca" pod "5e0cf260-9d68-4a89-9e8c-5c8f43496ec1" (UID: "5e0cf260-9d68-4a89-9e8c-5c8f43496ec1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:19.107661 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:19.107331 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-maas-api-xks-ca\") pod \"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1\" (UID: \"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1\") " Oct 31 01:00:19.107661 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:19.107414 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-kube-api-access-qqqkw\" (UniqueName: \"kubernetes.io/projected/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-kube-api-access-qqqkw\") pod \"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1\" (UID: \"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1\") " Oct 31 01:00:19.107661 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:19.107617 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-maas-api-openshift-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:19.109687 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:19.109624 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-kube-api-access-qqqkw" pod "5e0cf260-9d68-4a89-9e8c-5c8f43496ec1" (UID: "5e0cf260-9d68-4a89-9e8c-5c8f43496ec1"). InnerVolumeSpecName "kube-api-access-qqqkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:19.110137 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:19.110111 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-maas-api-xks-ca" pod "5e0cf260-9d68-4a89-9e8c-5c8f43496ec1" (UID: "5e0cf260-9d68-4a89-9e8c-5c8f43496ec1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:00:19.208578 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:19.208532 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-maas-api-xks-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:19.208578 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:19.208567 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqqkw\" (UniqueName: \"kubernetes.io/projected/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1-kube-api-access-qqqkw\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:19.711979 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:19.711934 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" event={"ID":"5e0cf260-9d68-4a89-9e8c-5c8f43496ec1","Type":"ContainerDied","Data":"9418e7ceb7cd0e0e099cac696834ed41259afbebbc28b34978286319cd4d5660"} Oct 31 01:00:19.712174 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:19.712005 4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9418e7ceb7cd0e0e099cac696834ed41259afbebbc28b34978286319cd4d5660" Oct 31 01:00:19.712174 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:19.712012 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt" Oct 31 01:00:26.334351 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:26.334316 4519 scope.go:122] "RemoveContainer" containerID="c686a347496ee0992bce8dec43054acd984410bb7fdf80b979de456db19047ae" Oct 31 01:00:26.342729 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:26.342694 4519 scope.go:122] "RemoveContainer" containerID="12efcdd749a03bc4071ef2d016e8d28fbcdd391d3d3934b418485691642aa63f" Oct 31 01:00:26.350354 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:26.350334 4519 scope.go:122] "RemoveContainer" containerID="a22e10eae35a8da38a266d6a5ce3c8745201a0dcab2dbb946bc206f3beead99b" Oct 31 01:00:26.642879 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:26.642787 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh"] Oct 31 01:00:26.747156 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:26.747116 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:28.122481 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:28.122452 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:41.681367 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:41.681305 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" podUID="96d7c60c-14c0-4287-8c28-88f17fbe205e" containerName="main" containerID="cri-o://4922cfb0f36b2f7c073c312e7164dab7bb52f44dc0371c7e868eefb3dab05e6c" gracePeriod=45 Oct 31 01:00:41.893383 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:41.893333 4519 generic.go:356] "Generic (PLEG): container finished" podID="96d7c60c-14c0-4287-8c28-88f17fbe205e" containerID="4922cfb0f36b2f7c073c312e7164dab7bb52f44dc0371c7e868eefb3dab05e6c" exitCode=0 Oct 31 01:00:41.893597 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:41.893420 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" event={"ID":"96d7c60c-14c0-4287-8c28-88f17fbe205e","Type":"ContainerDied","Data":"4922cfb0f36b2f7c073c312e7164dab7bb52f44dc0371c7e868eefb3dab05e6c"} Oct 31 01:00:42.166250 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.165864 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 01:00:42.295686 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.294957 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96d7c60c-14c0-4287-8c28-88f17fbe205e-tls-certs\" (UniqueName: \"kubernetes.io/secret/96d7c60c-14c0-4287-8c28-88f17fbe205e-tls-certs\") pod \"96d7c60c-14c0-4287-8c28-88f17fbe205e\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " Oct 31 01:00:42.295686 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.295019 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-dshm\") pod \"96d7c60c-14c0-4287-8c28-88f17fbe205e\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " Oct 31 01:00:42.295686 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.295092 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-tmp-dir\") pod \"96d7c60c-14c0-4287-8c28-88f17fbe205e\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " Oct 31 01:00:42.295686 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.295129 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-model-cache\") pod \"96d7c60c-14c0-4287-8c28-88f17fbe205e\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " Oct 31 01:00:42.295686 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.295191 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-home\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-home\") pod \"96d7c60c-14c0-4287-8c28-88f17fbe205e\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " Oct 31 01:00:42.295686 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.295259 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d7c60c-14c0-4287-8c28-88f17fbe205e-kube-api-access-t59xq\" (UniqueName: \"kubernetes.io/projected/96d7c60c-14c0-4287-8c28-88f17fbe205e-kube-api-access-t59xq\") pod \"96d7c60c-14c0-4287-8c28-88f17fbe205e\" (UID: \"96d7c60c-14c0-4287-8c28-88f17fbe205e\") " Oct 31 01:00:42.296539 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.296505 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-model-cache" pod "96d7c60c-14c0-4287-8c28-88f17fbe205e" (UID: "96d7c60c-14c0-4287-8c28-88f17fbe205e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:42.297320 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.297290 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-tmp-dir" pod "96d7c60c-14c0-4287-8c28-88f17fbe205e" (UID: "96d7c60c-14c0-4287-8c28-88f17fbe205e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:42.297539 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.297518 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-home" pod "96d7c60c-14c0-4287-8c28-88f17fbe205e" (UID: "96d7c60c-14c0-4287-8c28-88f17fbe205e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:42.300491 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.300451 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-dshm" pod "96d7c60c-14c0-4287-8c28-88f17fbe205e" (UID: "96d7c60c-14c0-4287-8c28-88f17fbe205e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:42.302847 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.302798 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d7c60c-14c0-4287-8c28-88f17fbe205e-kube-api-access-t59xq" pod "96d7c60c-14c0-4287-8c28-88f17fbe205e" (UID: "96d7c60c-14c0-4287-8c28-88f17fbe205e"). InnerVolumeSpecName "kube-api-access-t59xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:42.303126 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.303092 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d7c60c-14c0-4287-8c28-88f17fbe205e-tls-certs" pod "96d7c60c-14c0-4287-8c28-88f17fbe205e" (UID: "96d7c60c-14c0-4287-8c28-88f17fbe205e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:00:42.396324 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.396184 4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-dshm\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:42.396324 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.396227 4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-tmp-dir\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:42.396324 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.396245 4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-model-cache\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:42.396324 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.396261 4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d7c60c-14c0-4287-8c28-88f17fbe205e-home\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:42.396324 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.396277 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t59xq\" (UniqueName: \"kubernetes.io/projected/96d7c60c-14c0-4287-8c28-88f17fbe205e-kube-api-access-t59xq\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:42.396324 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.396292 4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d7c60c-14c0-4287-8c28-88f17fbe205e-tls-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:42.901813 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.901762 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" event={"ID":"96d7c60c-14c0-4287-8c28-88f17fbe205e","Type":"ContainerDied","Data":"583394cd99acaea1dbcf321668975ba85f06061f68d46061eba90637e8a4324c"} Oct 31 01:00:42.901813 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.901791 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh" Oct 31 01:00:42.902423 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.901833 4519 scope.go:122] "RemoveContainer" containerID="4922cfb0f36b2f7c073c312e7164dab7bb52f44dc0371c7e868eefb3dab05e6c" Oct 31 01:00:42.927035 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.927000 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh"] Oct 31 01:00:42.931151 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:42.931122 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2f32c431-kserve-6b9c94d59c-zcwgh"] Oct 31 01:00:44.101837 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:44.101790 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96d7c60c-14c0-4287-8c28-88f17fbe205e" path="/var/lib/kubelet/pods/96d7c60c-14c0-4287-8c28-88f17fbe205e/volumes" Oct 31 01:00:46.664345 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:46.664305 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:46.777991 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:00:46.777950 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:47.540968 ip-10-0-143-188 kubenswrapper[4519]: E1031 01:00:47.540929 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 01:01:02.306395 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.306350 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2"] Oct 31 01:01:02.306918 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.306883 4519 cpu_manager.go:414] "containerMap: removing container" podUID="96d7c60c-14c0-4287-8c28-88f17fbe205e" containerName="main" Oct 31 01:01:02.306918 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.306902 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d7c60c-14c0-4287-8c28-88f17fbe205e" containerName="main" Oct 31 01:01:02.306918 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.306916 4519 cpu_manager.go:414] "containerMap: removing container" podUID="5e0cf260-9d68-4a89-9e8c-5c8f43496ec1" containerName="revoke-keys" Oct 31 01:01:02.307100 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.306924 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e0cf260-9d68-4a89-9e8c-5c8f43496ec1" containerName="revoke-keys" Oct 31 01:01:02.307100 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.307019 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d7c60c-14c0-4287-8c28-88f17fbe205e" containerName="main" Oct 31 01:01:02.307100 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.307034 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e0cf260-9d68-4a89-9e8c-5c8f43496ec1" containerName="revoke-keys" Oct 31 01:01:02.310216 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.310194 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:02.320437 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.320395 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 01:01:02.320821 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.320531 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:01:02.321016 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.320576 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 01:01:02.321016 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.320668 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:01:02.321016 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.320734 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xds7h\"" Oct 31 01:01:02.322806 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.322777 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2"] Oct 31 01:01:02.428424 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.428380 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6300af0-1821-4399-a54d-1ef9ffd96dbd-openshift-service-ca\") pod \"maas-api-f8cb46844-cb6j2\" (UID: \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\") " pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:02.428673 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.428548 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6300af0-1821-4399-a54d-1ef9ffd96dbd-metrics-certs\") pod \"maas-api-f8cb46844-cb6j2\" (UID: \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\") " pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:02.428673 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.428584 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89gbb\" (UniqueName: \"kubernetes.io/projected/c6300af0-1821-4399-a54d-1ef9ffd96dbd-kube-api-access-89gbb\") pod \"maas-api-f8cb46844-cb6j2\" (UID: \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\") " pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:02.428673 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.428610 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6300af0-1821-4399-a54d-1ef9ffd96dbd-maas-api-tls\") pod \"maas-api-f8cb46844-cb6j2\" (UID: \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\") " pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:02.480220 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.480188 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:02.529213 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.529167 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6300af0-1821-4399-a54d-1ef9ffd96dbd-metrics-certs\") pod \"maas-api-f8cb46844-cb6j2\" (UID: \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\") " pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:02.529213 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.529215 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89gbb\" (UniqueName: \"kubernetes.io/projected/c6300af0-1821-4399-a54d-1ef9ffd96dbd-kube-api-access-89gbb\") pod \"maas-api-f8cb46844-cb6j2\" (UID: \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\") " pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:02.529493 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.529236 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6300af0-1821-4399-a54d-1ef9ffd96dbd-maas-api-tls\") pod \"maas-api-f8cb46844-cb6j2\" (UID: \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\") " pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:02.529493 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.529259 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6300af0-1821-4399-a54d-1ef9ffd96dbd-openshift-service-ca\") pod \"maas-api-f8cb46844-cb6j2\" (UID: \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\") " pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:02.529988 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.529965 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6300af0-1821-4399-a54d-1ef9ffd96dbd-openshift-service-ca\") pod \"maas-api-f8cb46844-cb6j2\" (UID: \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\") " pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:02.531756 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.531734 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6300af0-1821-4399-a54d-1ef9ffd96dbd-maas-api-tls\") pod \"maas-api-f8cb46844-cb6j2\" (UID: \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\") " pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:02.531855 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.531776 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6300af0-1821-4399-a54d-1ef9ffd96dbd-metrics-certs\") pod \"maas-api-f8cb46844-cb6j2\" (UID: \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\") " pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:02.538421 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.538397 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89gbb\" (UniqueName: \"kubernetes.io/projected/c6300af0-1821-4399-a54d-1ef9ffd96dbd-kube-api-access-89gbb\") pod \"maas-api-f8cb46844-cb6j2\" (UID: \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\") " pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:02.624847 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.624760 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:02.899892 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.899807 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2"] Oct 31 01:01:02.902088 ip-10-0-143-188 kubenswrapper[4519]: W1031 01:01:02.902058 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6300af0_1821_4399_a54d_1ef9ffd96dbd.slice/crio-b02e964cad46225d71986e7d6dfa1f435be59a45925886da50bad4e3b21c0ea0 WatchSource:0}: Error finding container b02e964cad46225d71986e7d6dfa1f435be59a45925886da50bad4e3b21c0ea0: Status 404 returned error can't find the container with id b02e964cad46225d71986e7d6dfa1f435be59a45925886da50bad4e3b21c0ea0 Oct 31 01:01:02.903417 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.903400 4519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:01:02.983129 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:02.983091 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" event={"ID":"c6300af0-1821-4399-a54d-1ef9ffd96dbd","Type":"ContainerStarted","Data":"b02e964cad46225d71986e7d6dfa1f435be59a45925886da50bad4e3b21c0ea0"} Oct 31 01:01:03.988715 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:03.988681 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" event={"ID":"c6300af0-1821-4399-a54d-1ef9ffd96dbd","Type":"ContainerStarted","Data":"971652b4bf54a2048bc7e0b49d2e5c556857b0f5f8de4b985f422b0098a0c892"} Oct 31 01:01:03.989121 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:03.988747 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:04.009740 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:04.009625 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" podStartSLOduration=1.52901647 podStartE2EDuration="2.009604941s" totalImagesPullingTime="480.588471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:04.00914147 +0000 UTC m=+1660.038321859" watchObservedRunningTime="2026-10-31 01:01:04.009604941 +0000 UTC m=+1660.038785331" Oct 31 01:01:09.335923 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:09.335825 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:09.846936 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:09.846887 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:09.998117 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:09.998085 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:10.066141 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:10.065600 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:10.434054 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:10.434019 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95"] Oct 31 01:01:10.437471 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:10.437450 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" Oct 31 01:01:10.447282 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:10.447242 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95"] Oct 31 01:01:10.496521 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:10.496478 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c860fbac-a728-4be9-9811-bc43360f3422-openshift-service-ca\") pod \"maas-api-97ff44f5b-g9p95\" (UID: \"c860fbac-a728-4be9-9811-bc43360f3422\") " pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" Oct 31 01:01:10.496720 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:10.496530 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqdw8\" (UniqueName: \"kubernetes.io/projected/c860fbac-a728-4be9-9811-bc43360f3422-kube-api-access-fqdw8\") pod \"maas-api-97ff44f5b-g9p95\" (UID: \"c860fbac-a728-4be9-9811-bc43360f3422\") " pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" Oct 31 01:01:10.496720 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:10.496607 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c860fbac-a728-4be9-9811-bc43360f3422-metrics-certs\") pod \"maas-api-97ff44f5b-g9p95\" (UID: \"c860fbac-a728-4be9-9811-bc43360f3422\") " pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" Oct 31 01:01:10.496720 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:10.496664 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c860fbac-a728-4be9-9811-bc43360f3422-maas-api-tls\") pod \"maas-api-97ff44f5b-g9p95\" (UID: \"c860fbac-a728-4be9-9811-bc43360f3422\") " pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" Oct 31 01:01:10.597706 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:10.597628 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqdw8\" (UniqueName: \"kubernetes.io/projected/c860fbac-a728-4be9-9811-bc43360f3422-kube-api-access-fqdw8\") pod \"maas-api-97ff44f5b-g9p95\" (UID: \"c860fbac-a728-4be9-9811-bc43360f3422\") " pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" Oct 31 01:01:10.597706 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:10.597704 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c860fbac-a728-4be9-9811-bc43360f3422-metrics-certs\") pod \"maas-api-97ff44f5b-g9p95\" (UID: \"c860fbac-a728-4be9-9811-bc43360f3422\") " pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" Oct 31 01:01:10.597977 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:10.597725 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c860fbac-a728-4be9-9811-bc43360f3422-maas-api-tls\") pod \"maas-api-97ff44f5b-g9p95\" (UID: \"c860fbac-a728-4be9-9811-bc43360f3422\") " pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" Oct 31 01:01:10.597977 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:10.597808 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c860fbac-a728-4be9-9811-bc43360f3422-openshift-service-ca\") pod \"maas-api-97ff44f5b-g9p95\" (UID: \"c860fbac-a728-4be9-9811-bc43360f3422\") " pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" Oct 31 01:01:10.598556 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:10.598527 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c860fbac-a728-4be9-9811-bc43360f3422-openshift-service-ca\") pod \"maas-api-97ff44f5b-g9p95\" (UID: \"c860fbac-a728-4be9-9811-bc43360f3422\") " pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" Oct 31 01:01:10.600286 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:10.600254 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c860fbac-a728-4be9-9811-bc43360f3422-metrics-certs\") pod \"maas-api-97ff44f5b-g9p95\" (UID: \"c860fbac-a728-4be9-9811-bc43360f3422\") " pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" Oct 31 01:01:10.600406 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:10.600311 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c860fbac-a728-4be9-9811-bc43360f3422-maas-api-tls\") pod \"maas-api-97ff44f5b-g9p95\" (UID: \"c860fbac-a728-4be9-9811-bc43360f3422\") " pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" Oct 31 01:01:10.612684 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:10.612652 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqdw8\" (UniqueName: \"kubernetes.io/projected/c860fbac-a728-4be9-9811-bc43360f3422-kube-api-access-fqdw8\") pod \"maas-api-97ff44f5b-g9p95\" (UID: \"c860fbac-a728-4be9-9811-bc43360f3422\") " pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" Oct 31 01:01:10.751158 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:10.751118 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" Oct 31 01:01:11.028784 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:11.028701 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95"] Oct 31 01:01:11.031086 ip-10-0-143-188 kubenswrapper[4519]: W1031 01:01:11.031055 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc860fbac_a728_4be9_9811_bc43360f3422.slice/crio-b92b968d7c94aade41328f2fe032e34fe8fac12438f2f68f3b0f0469680f3b17 WatchSource:0}: Error finding container b92b968d7c94aade41328f2fe032e34fe8fac12438f2f68f3b0f0469680f3b17: Status 404 returned error can't find the container with id b92b968d7c94aade41328f2fe032e34fe8fac12438f2f68f3b0f0469680f3b17 Oct 31 01:01:12.023743 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:12.023702 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" event={"ID":"c860fbac-a728-4be9-9811-bc43360f3422","Type":"ContainerStarted","Data":"16a019e506627ac07aeb06c930dc5294dc5895638b375528c5adbd5671cd0a8c"} Oct 31 01:01:12.024170 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:12.023770 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" event={"ID":"c860fbac-a728-4be9-9811-bc43360f3422","Type":"ContainerStarted","Data":"b92b968d7c94aade41328f2fe032e34fe8fac12438f2f68f3b0f0469680f3b17"} Oct 31 01:01:12.024170 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:12.023915 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" Oct 31 01:01:12.043890 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:12.043842 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" podStartSLOduration=1.553250926 podStartE2EDuration="2.043829271s" totalImagesPullingTime="490.578345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:12.04331094 +0000 UTC m=+1668.072491345" watchObservedRunningTime="2026-10-31 01:01:12.043829271 +0000 UTC m=+1668.073009673" Oct 31 01:01:12.228151 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:12.228089 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:12.938452 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:12.938401 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:13.036056 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:13.036019 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:14.170307 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:14.170269 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:14.269079 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:14.269015 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:14.364215 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:14.364178 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:14.879549 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:14.879503 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:15.663303 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:15.663274 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:16.566198 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:16.566144 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:17.267945 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:17.267911 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:17.541213 ip-10-0-143-188 kubenswrapper[4519]: E1031 01:01:17.541122 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 01:01:17.935122 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:17.935001 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:18.036044 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:18.036018 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-97ff44f5b-g9p95" Oct 31 01:01:18.089010 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:18.088976 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2"] Oct 31 01:01:18.089272 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:18.089247 4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" podUID="c6300af0-1821-4399-a54d-1ef9ffd96dbd" containerName="maas-api" containerID="cri-o://971652b4bf54a2048bc7e0b49d2e5c556857b0f5f8de4b985f422b0098a0c892" gracePeriod=30 Oct 31 01:01:18.336660 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:18.336617 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:18.370522 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:18.370483 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6300af0-1821-4399-a54d-1ef9ffd96dbd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6300af0-1821-4399-a54d-1ef9ffd96dbd-maas-api-tls\") pod \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\" (UID: \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\") " Oct 31 01:01:18.370735 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:18.370566 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6300af0-1821-4399-a54d-1ef9ffd96dbd-kube-api-access-89gbb\" (UniqueName: \"kubernetes.io/projected/c6300af0-1821-4399-a54d-1ef9ffd96dbd-kube-api-access-89gbb\") pod \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\" (UID: \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\") " Oct 31 01:01:18.370735 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:18.370672 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6300af0-1821-4399-a54d-1ef9ffd96dbd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6300af0-1821-4399-a54d-1ef9ffd96dbd-metrics-certs\") pod \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\" (UID: \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\") " Oct 31 01:01:18.370876 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:18.370747 4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6300af0-1821-4399-a54d-1ef9ffd96dbd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6300af0-1821-4399-a54d-1ef9ffd96dbd-openshift-service-ca\") pod \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\" (UID: \"c6300af0-1821-4399-a54d-1ef9ffd96dbd\") " Oct 31 01:01:18.371126 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:18.371102 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6300af0-1821-4399-a54d-1ef9ffd96dbd-openshift-service-ca" pod "c6300af0-1821-4399-a54d-1ef9ffd96dbd" (UID: "c6300af0-1821-4399-a54d-1ef9ffd96dbd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:18.372735 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:18.372700 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6300af0-1821-4399-a54d-1ef9ffd96dbd-maas-api-tls" pod "c6300af0-1821-4399-a54d-1ef9ffd96dbd" (UID: "c6300af0-1821-4399-a54d-1ef9ffd96dbd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:18.372835 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:18.372750 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6300af0-1821-4399-a54d-1ef9ffd96dbd-kube-api-access-89gbb" pod "c6300af0-1821-4399-a54d-1ef9ffd96dbd" (UID: "c6300af0-1821-4399-a54d-1ef9ffd96dbd"). InnerVolumeSpecName "kube-api-access-89gbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:18.372876 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:18.372829 4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6300af0-1821-4399-a54d-1ef9ffd96dbd-metrics-certs" pod "c6300af0-1821-4399-a54d-1ef9ffd96dbd" (UID: "c6300af0-1821-4399-a54d-1ef9ffd96dbd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:18.472201 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:18.472113 4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89gbb\" (UniqueName: \"kubernetes.io/projected/c6300af0-1821-4399-a54d-1ef9ffd96dbd-kube-api-access-89gbb\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:18.472201 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:18.472143 4519 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6300af0-1821-4399-a54d-1ef9ffd96dbd-metrics-certs\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:18.472201 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:18.472153 4519 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6300af0-1821-4399-a54d-1ef9ffd96dbd-openshift-service-ca\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:18.472201 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:18.472161 4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6300af0-1821-4399-a54d-1ef9ffd96dbd-maas-api-tls\") on node \"ip-10-0-143-188.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:18.764165 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:18.764130 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:19.055476 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:19.055392 4519 generic.go:356] "Generic (PLEG): container finished" podID="c6300af0-1821-4399-a54d-1ef9ffd96dbd" containerID="971652b4bf54a2048bc7e0b49d2e5c556857b0f5f8de4b985f422b0098a0c892" exitCode=0 Oct 31 01:01:19.055476 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:19.055445 4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" Oct 31 01:01:19.055476 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:19.055465 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" event={"ID":"c6300af0-1821-4399-a54d-1ef9ffd96dbd","Type":"ContainerDied","Data":"971652b4bf54a2048bc7e0b49d2e5c556857b0f5f8de4b985f422b0098a0c892"} Oct 31 01:01:19.055738 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:19.055517 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2" event={"ID":"c6300af0-1821-4399-a54d-1ef9ffd96dbd","Type":"ContainerDied","Data":"b02e964cad46225d71986e7d6dfa1f435be59a45925886da50bad4e3b21c0ea0"} Oct 31 01:01:19.055738 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:19.055547 4519 scope.go:122] "RemoveContainer" containerID="971652b4bf54a2048bc7e0b49d2e5c556857b0f5f8de4b985f422b0098a0c892" Oct 31 01:01:19.064562 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:19.064542 4519 scope.go:122] "RemoveContainer" containerID="971652b4bf54a2048bc7e0b49d2e5c556857b0f5f8de4b985f422b0098a0c892" Oct 31 01:01:19.064858 ip-10-0-143-188 kubenswrapper[4519]: E1031 01:01:19.064837 4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"971652b4bf54a2048bc7e0b49d2e5c556857b0f5f8de4b985f422b0098a0c892\": container with ID starting with 971652b4bf54a2048bc7e0b49d2e5c556857b0f5f8de4b985f422b0098a0c892 not found: ID does not exist" containerID="971652b4bf54a2048bc7e0b49d2e5c556857b0f5f8de4b985f422b0098a0c892" Oct 31 01:01:19.064937 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:19.064866 4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"971652b4bf54a2048bc7e0b49d2e5c556857b0f5f8de4b985f422b0098a0c892"} err="failed to get container status \"971652b4bf54a2048bc7e0b49d2e5c556857b0f5f8de4b985f422b0098a0c892\": rpc error: code = NotFound desc = could not find container \"971652b4bf54a2048bc7e0b49d2e5c556857b0f5f8de4b985f422b0098a0c892\": container with ID starting with 971652b4bf54a2048bc7e0b49d2e5c556857b0f5f8de4b985f422b0098a0c892 not found: ID does not exist" Oct 31 01:01:19.077422 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:19.077389 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2"] Oct 31 01:01:19.081670 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:19.081626 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-f8cb46844-cb6j2"] Oct 31 01:01:19.500954 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:19.500915 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:20.100807 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:20.100769 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6300af0-1821-4399-a54d-1ef9ffd96dbd" path="/var/lib/kubelet/pods/c6300af0-1821-4399-a54d-1ef9ffd96dbd/volumes" Oct 31 01:01:20.156181 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:20.156151 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:20.847248 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:20.847212 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:21.556983 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:21.556953 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:22.312647 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:22.312594 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:22.938961 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:22.938923 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:23.545282 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:23.545250 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:24.349227 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:24.349192 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:25.046111 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:25.046067 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:25.740334 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:25.740298 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:26.345091 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:26.345048 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:27.052234 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:27.052198 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:27.833954 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:27.833922 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:28.453659 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:28.453610 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:29.144298 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:29.144262 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:29.747712 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:29.747676 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:30.356546 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:30.356510 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:30.949572 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:30.949523 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:31.620524 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:31.620464 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:32.239249 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:32.239216 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:32.850930 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:32.850890 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:33.547623 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:33.547590 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:33.763865 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:33.763833 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:33.863501 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:33.863409 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:34.350783 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:34.350740 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:34.796857 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:34.796822 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:34.921422 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:34.921393 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:35.027697 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:35.027666 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:40.063618 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:40.063587 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:41.365875 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:41.365837 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:41.814361 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:41.814317 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:41.921667 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:41.919255 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:44.771802 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:44.771769 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:44.865474 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:44.865440 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:46.028448 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:46.028416 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:46.134656 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:46.134597 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:46.232706 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:46.232668 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:46.847399 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:46.847370 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:47.542091 ip-10-0-143-188 kubenswrapper[4519]: E1031 01:01:47.542050 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 01:01:47.639089 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:47.639057 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:48.245859 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:48.245813 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:48.950306 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:48.950274 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:49.636590 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:49.636553 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:50.257366 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:50.257329 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:50.955907 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:50.955873 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:51.555324 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:51.555285 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:52.154546 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:52.154513 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:52.840569 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:52.840517 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:53.495191 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:53.495152 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:54.147668 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:54.147621 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:54.796013 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:54.795980 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:55.677661 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:55.677607 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:56.343870 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:56.343832 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:56.953387 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:56.953350 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:57.558669 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:57.558617 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:58.252554 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:58.252523 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:58.984010 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:58.983964 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:59.540652 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:59.540599 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:59.752199 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:01:59.752165 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:00.135699 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:00.135669 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:00.358217 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:00.358181 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:01.039028 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:01.038993 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:01.896446 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:01.896370 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:02.450197 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:02.450161 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:03.157028 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:03.156992 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:03.954252 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:03.954218 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:04.638711 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:04.638680 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:05.330237 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:05.330198 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:06.026400 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:06.026364 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:06.681600 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:06.681567 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:07.553363 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:07.553330 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:08.268073 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:08.268043 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:08.937297 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:08.937256 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:09.585336 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:09.585304 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:10.258526 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:10.258483 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:10.965066 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:10.965030 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:11.653611 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:11.653581 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:12.273730 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:12.273695 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:12.374089 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:12.374051 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:13.036126 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:13.036094 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:13.751290 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:13.751257 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:14.532036 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:14.532003 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:15.253031 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:15.252993 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:15.953961 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:15.953926 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:16.559047 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:16.559012 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:17.258153 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:17.258117 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:17.541529 ip-10-0-143-188 kubenswrapper[4519]: E1031 01:02:17.541447 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 01:02:17.960667 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:17.960551 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:18.645445 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:18.645416 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:19.119127 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:19.119086 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:19.192478 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:19.192442 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:19.272808 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:19.272777 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:19.910416 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:19.910382 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:20.556020 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:20.555989 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:21.165146 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:21.165110 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:21.833437 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:21.833390 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:22.555320 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:22.555288 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:23.163089 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:23.163051 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:23.848285 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:23.848249 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:24.670202 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:24.670167 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:25.270849 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:25.270811 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:25.978124 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:25.978089 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:26.643062 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:26.643025 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:27.259328 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:27.259286 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:28.052792 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:28.052747 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:28.655661 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:28.655601 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:29.254101 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:29.254058 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:29.896579 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:29.896544 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:30.532256 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:30.532227 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:31.159382 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:31.159351 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:31.925831 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:31.925798 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:32.545456 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:32.545424 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:33.217934 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:33.217904 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:33.855357 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:33.855322 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:34.542178 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:34.542144 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:35.253300 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:35.253263 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:35.935725 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:35.935648 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:36.591230 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:36.591194 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:37.241773 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:37.241736 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:37.842383 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:37.842343 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:38.548107 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:38.548064 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:39.164816 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:39.164783 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:39.826302 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:39.826264 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:40.545270 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:40.545239 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:41.145664 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:41.145621 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:41.743245 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:41.743213 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:42.344730 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:42.344620 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:42.942532 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:42.942491 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:43.543109 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:43.543073 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:44.142587 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:44.142552 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:44.754580 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:44.754543 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:45.362455 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:45.362417 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:46.048947 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:46.048907 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:46.833880 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:46.833846 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:47.502041 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:47.502006 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:47.541022 ip-10-0-143-188 kubenswrapper[4519]: E1031 01:02:47.540988 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 01:02:48.146996 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:48.146960 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:48.751111 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:48.751054 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:49.367387 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:49.367356 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:50.037181 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:50.037149 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:50.653793 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:50.653757 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:51.302043 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:51.302005 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:51.933210 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:51.933174 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:52.633490 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:52.633460 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:53.252649 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:53.252589 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:53.847569 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:53.847530 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:54.545600 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:54.545568 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:55.155552 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:55.155518 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:55.753292 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:55.753258 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:56.382253 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:56.382210 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:57.038189 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:57.038154 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:57.741918 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:57.741881 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:58.360622 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:58.360590 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:59.052203 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:59.052151 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:59.666109 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:02:59.666074 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:00.250340 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:00.250305 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:00.847055 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:00.847021 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:01.543594 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:01.543556 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:02.227882 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:02.227845 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:02.836143 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:02.836115 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:03.450699 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:03.450660 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:04.156696 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:04.156658 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:04.925329 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:04.925295 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:05.662338 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:05.662294 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:06.357127 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:06.357085 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:07.023466 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:07.023432 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:07.643290 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:07.643258 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:08.294170 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:08.294116 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:08.840012 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:08.839967 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:09.643918 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:09.643882 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:10.266982 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:10.266952 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:10.864552 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:10.864514 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:11.649064 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:11.649026 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:12.364988 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:12.364957 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:12.459780 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:12.459732 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:13.136520 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:13.136470 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:13.755355 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:13.755322 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:14.417844 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:14.417801 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:15.046232 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:15.046197 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:15.758235 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:15.758200 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:16.416413 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:16.416366 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:17.083444 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:17.083402 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:17.541041 ip-10-0-143-188 kubenswrapper[4519]: E1031 01:03:17.541003 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 01:03:17.757842 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:17.757790 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:18.459268 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:18.459220 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:19.052451 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:19.052413 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:19.731381 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:19.731345 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:20.436511 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:20.436475 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:21.461907 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:21.461857 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:21.847191 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:21.847155 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:22.501079 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:22.501050 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:23.228954 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:23.228916 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:23.844815 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:23.844775 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:24.533387 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:24.533351 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:25.147719 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:25.147630 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:25.834138 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:25.834093 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:26.525916 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:26.525877 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:27.151217 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:27.151182 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:27.853263 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:27.853232 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:28.457292 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:28.457242 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:29.138038 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:29.137992 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:29.742204 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:29.742170 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:30.011349 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:30.011251 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:30.105954 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:30.105918 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:35.295084 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:35.295039 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:35.425451 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:35.425412 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:35.520061 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:35.520018 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:35.972134 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:35.972096 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:41.741508 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:41.741470 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:41.824878 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:41.824842 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:42.065730 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:03:42.065652 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:47.541725 ip-10-0-143-188 kubenswrapper[4519]: E1031 01:03:47.541677 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 01:04:09.380925 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:04:09.380781 4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-457nv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:04:09.381825 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:04:09.381767 4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-457nv" podUID="23d1c38d-9d1d-4674-82cc-38782f2a4f39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:04:12.023410 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:04:12.023376 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2"] Oct 31 01:04:12.024745 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:04:12.024722 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e0c5581-vvpn2"] Oct 31 01:04:12.100199 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:04:12.100164 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6251df92-be00-4d76-8346-3ce0c4c9f528" path="/var/lib/kubelet/pods/6251df92-be00-4d76-8346-3ce0c4c9f528/volumes" Oct 31 01:04:12.401888 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:04:12.401805 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:17.539904 ip-10-0-143-188 kubenswrapper[4519]: E1031 01:04:17.539861 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 01:04:44.436205 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:04:44.436167 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8kw7h_43e494ff-c9a1-4707-8f27-efd46b218d97/manager/0.log" Oct 31 01:04:44.629195 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:04:44.629159 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-hzz82_3147d13d-fa94-4a4c-a468-b7c9e275ef43/manager/0.log" Oct 31 01:04:45.505219 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:04:45.505188 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-hkvls_c0bb009c-f110-4275-8030-46207de5b3e2/proxy/0.log" Oct 31 01:04:46.259919 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:04:46.259890 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-97ff44f5b-g9p95_c860fbac-a728-4be9-9811-bc43360f3422/maas-api/0.log" Oct 31 01:04:47.215699 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:04:47.215669 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-gnzpt_5e0cf260-9d68-4a89-9e8c-5c8f43496ec1/revoke-keys/0.log" Oct 31 01:04:47.543669 ip-10-0-143-188 kubenswrapper[4519]: E1031 01:04:47.543611 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 01:04:51.447043 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:04:51.447008 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-nnw2v_18daa545-6a52-4ba9-af4b-b60b4ac1f98c/discovery/0.log" Oct 31 01:04:51.827774 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:04:51.827738 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b544748f5-zkdsq_d7213fb5-28f3-41c1-bd33-4e922f88120d/kube-auth-proxy/0.log" Oct 31 01:04:51.995556 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:04:51.995522 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-457nv_23d1c38d-9d1d-4674-82cc-38782f2a4f39/istio-proxy/0.log" Oct 31 01:04:52.366839 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:04:52.366795 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-8wb6m_a11bc6fe-fa8b-425e-b6df-007d3fc6aa07/istio-proxy/0.log" Oct 31 01:04:52.753680 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:04:52.753651 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-ggdf4_1a6c37d5-3025-470f-94c0-ee1502fc7a76/istio-proxy/0.log" Oct 31 01:05:07.124341 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:05:07.124302 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8kw7h_43e494ff-c9a1-4707-8f27-efd46b218d97/manager/0.log" Oct 31 01:05:07.317363 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:05:07.317328 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-hzz82_3147d13d-fa94-4a4c-a468-b7c9e275ef43/manager/0.log" Oct 31 01:05:08.214202 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:05:08.214166 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-hkvls_c0bb009c-f110-4275-8030-46207de5b3e2/proxy/0.log" Oct 31 01:05:08.819951 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:05:08.819916 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-97ff44f5b-g9p95_c860fbac-a728-4be9-9811-bc43360f3422/maas-api/0.log" Oct 31 01:05:09.839955 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:05:09.839925 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-gnzpt_5e0cf260-9d68-4a89-9e8c-5c8f43496ec1/revoke-keys/0.log" Oct 31 01:05:12.454097 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:05:12.454053 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:13.677088 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:05:13.677050 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-nnw2v_18daa545-6a52-4ba9-af4b-b60b4ac1f98c/discovery/0.log" Oct 31 01:05:14.062710 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:05:14.062680 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b544748f5-zkdsq_d7213fb5-28f3-41c1-bd33-4e922f88120d/kube-auth-proxy/0.log" Oct 31 01:05:14.257620 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:05:14.257588 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-457nv_23d1c38d-9d1d-4674-82cc-38782f2a4f39/istio-proxy/0.log" Oct 31 01:05:14.686931 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:05:14.686900 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-8wb6m_a11bc6fe-fa8b-425e-b6df-007d3fc6aa07/istio-proxy/0.log" Oct 31 01:05:15.069949 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:05:15.069920 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-ggdf4_1a6c37d5-3025-470f-94c0-ee1502fc7a76/istio-proxy/0.log" Oct 31 01:05:17.542001 ip-10-0-143-188 kubenswrapper[4519]: E1031 01:05:17.541962 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 01:05:19.020486 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:05:19.020450 4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt"] Oct 31 01:05:19.022943 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:05:19.022908 4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gnzpt"] Oct 31 01:05:20.100391 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:05:20.100357 4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e0cf260-9d68-4a89-9e8c-5c8f43496ec1" path="/var/lib/kubelet/pods/5e0cf260-9d68-4a89-9e8c-5c8f43496ec1/volumes" Oct 31 01:05:26.410580 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:05:26.410546 4519 scope.go:122] "RemoveContainer" containerID="0f2f2e9ad3065840127102fa4f5cf987c941396279e2f218e20d30280efca0df" Oct 31 01:05:26.418812 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:05:26.418747 4519 scope.go:122] "RemoveContainer" containerID="748907ab8512d4f0246a87a0b583e678cc191c39072eefba071c70106d41d449" Oct 31 01:05:47.541258 ip-10-0-143-188 kubenswrapper[4519]: E1031 01:05:47.541171 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 01:06:12.524159 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:12.524100 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:17.541676 ip-10-0-143-188 kubenswrapper[4519]: E1031 01:06:17.541616 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 01:06:26.032226 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:26.032188 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:26.447263 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:26.447177 4519 scope.go:122] "RemoveContainer" containerID="b7ea4d52a9f79a95918c112cc47e349f16bd2bd8025727b4b0b979fe6dbc607e" Oct 31 01:06:44.665378 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:44.665341 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:44.742583 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:44.742545 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:47.541397 ip-10-0-143-188 kubenswrapper[4519]: E1031 01:06:47.541356 4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgt89" containerName="extractor" Oct 31 01:06:48.003721 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:48.003690 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7tgpr_ffd93b08-b857-4571-a1b5-5eaca4456f8b/global-pull-secret-syncer/0.log" Oct 31 01:06:48.179356 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:48.179319 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" Oct 31 01:06:48.322270 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:48.322192 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-188.us-east-2.compute.internal_c7fa48eaf6754635db0f21b009865b16/haproxy/0.log" Oct 31 01:06:57.663350 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:57.663319 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-d969cdb45-lvjhp_76a22588-6b4d-4c35-b00a-eaae2cff3ae7/metrics-server/0.log" Oct 31 01:06:57.784148 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:57.784118 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fmstj_ee44fef7-b277-4895-8ba0-def045d88a5d/node-exporter/0.log" Oct 31 01:06:57.807250 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:57.807206 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fmstj_ee44fef7-b277-4895-8ba0-def045d88a5d/kube-rbac-proxy/0.log" Oct 31 01:06:57.827814 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:57.827786 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fmstj_ee44fef7-b277-4895-8ba0-def045d88a5d/init-textfile/0.log" Oct 31 01:06:58.234489 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.234460 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-s68fl_204a65d0-1d9e-44b9-bbd5-86142f691f51/prometheus-operator-admission-webhook/0.log" Oct 31 01:06:58.331322 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.331282 4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8"] Oct 31 01:06:58.331826 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.331805 4519 cpu_manager.go:414] "containerMap: removing container" podUID="c6300af0-1821-4399-a54d-1ef9ffd96dbd" containerName="maas-api" Oct 31 01:06:58.331826 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.331825 4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6300af0-1821-4399-a54d-1ef9ffd96dbd" containerName="maas-api" Oct 31 01:06:58.332002 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.331890 4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6300af0-1821-4399-a54d-1ef9ffd96dbd" containerName="maas-api" Oct 31 01:06:58.334901 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.334878 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:58.344292 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.344253 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8"] Oct 31 01:06:58.362477 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.362438 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8296280-5dd7-4049-9513-55d519d0230d-proc\") pod \"perf-node-gather-daemonset-drhn8\" (UID: \"d8296280-5dd7-4049-9513-55d519d0230d\") " pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:58.362477 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.362476 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8296280-5dd7-4049-9513-55d519d0230d-lib-modules\") pod \"perf-node-gather-daemonset-drhn8\" (UID: \"d8296280-5dd7-4049-9513-55d519d0230d\") " pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:58.362755 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.362579 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8296280-5dd7-4049-9513-55d519d0230d-podres\") pod \"perf-node-gather-daemonset-drhn8\" (UID: \"d8296280-5dd7-4049-9513-55d519d0230d\") " pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:58.362755 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.362626 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccvvd\" (UniqueName: \"kubernetes.io/projected/d8296280-5dd7-4049-9513-55d519d0230d-kube-api-access-ccvvd\") pod \"perf-node-gather-daemonset-drhn8\" (UID: \"d8296280-5dd7-4049-9513-55d519d0230d\") " pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:58.362755 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.362731 4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8296280-5dd7-4049-9513-55d519d0230d-sys\") pod \"perf-node-gather-daemonset-drhn8\" (UID: \"d8296280-5dd7-4049-9513-55d519d0230d\") " pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:58.428587 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.428553 4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bd47j\"/\"default-dockercfg-zkc7m\"" Oct 31 01:06:58.429659 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.429619 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bd47j\"/\"openshift-service-ca.crt\"" Oct 31 01:06:58.429774 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.429731 4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bd47j\"/\"kube-root-ca.crt\"" Oct 31 01:06:58.463697 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.463665 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8296280-5dd7-4049-9513-55d519d0230d-proc\") pod \"perf-node-gather-daemonset-drhn8\" (UID: \"d8296280-5dd7-4049-9513-55d519d0230d\") " pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:58.463804 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.463702 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8296280-5dd7-4049-9513-55d519d0230d-lib-modules\") pod \"perf-node-gather-daemonset-drhn8\" (UID: \"d8296280-5dd7-4049-9513-55d519d0230d\") " pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:58.463804 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.463759 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8296280-5dd7-4049-9513-55d519d0230d-podres\") pod \"perf-node-gather-daemonset-drhn8\" (UID: \"d8296280-5dd7-4049-9513-55d519d0230d\") " pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:58.463804 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.463781 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccvvd\" (UniqueName: \"kubernetes.io/projected/d8296280-5dd7-4049-9513-55d519d0230d-kube-api-access-ccvvd\") pod \"perf-node-gather-daemonset-drhn8\" (UID: \"d8296280-5dd7-4049-9513-55d519d0230d\") " pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:58.463959 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.463796 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8296280-5dd7-4049-9513-55d519d0230d-proc\") pod \"perf-node-gather-daemonset-drhn8\" (UID: \"d8296280-5dd7-4049-9513-55d519d0230d\") " pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:58.463959 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.463805 4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8296280-5dd7-4049-9513-55d519d0230d-sys\") pod \"perf-node-gather-daemonset-drhn8\" (UID: \"d8296280-5dd7-4049-9513-55d519d0230d\") " pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:58.463959 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.463862 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8296280-5dd7-4049-9513-55d519d0230d-podres\") pod \"perf-node-gather-daemonset-drhn8\" (UID: \"d8296280-5dd7-4049-9513-55d519d0230d\") " pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:58.463959 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.463910 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8296280-5dd7-4049-9513-55d519d0230d-sys\") pod \"perf-node-gather-daemonset-drhn8\" (UID: \"d8296280-5dd7-4049-9513-55d519d0230d\") " pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:58.463959 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.463915 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8296280-5dd7-4049-9513-55d519d0230d-lib-modules\") pod \"perf-node-gather-daemonset-drhn8\" (UID: \"d8296280-5dd7-4049-9513-55d519d0230d\") " pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:58.472360 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.472332 4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccvvd\" (UniqueName: \"kubernetes.io/projected/d8296280-5dd7-4049-9513-55d519d0230d-kube-api-access-ccvvd\") pod \"perf-node-gather-daemonset-drhn8\" (UID: \"d8296280-5dd7-4049-9513-55d519d0230d\") " pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:58.649855 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.649768 4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:58.923026 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:58.922923 4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8"] Oct 31 01:06:58.925301 ip-10-0-143-188 kubenswrapper[4519]: W1031 01:06:58.925265 4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd8296280_5dd7_4049_9513_55d519d0230d.slice/crio-7c91258422a875613d05f12cb15318e52f6dcb32a3b20e4f5ef14f07fff00672 WatchSource:0}: Error finding container 7c91258422a875613d05f12cb15318e52f6dcb32a3b20e4f5ef14f07fff00672: Status 404 returned error can't find the container with id 7c91258422a875613d05f12cb15318e52f6dcb32a3b20e4f5ef14f07fff00672 Oct 31 01:06:59.564351 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:59.564310 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" event={"ID":"d8296280-5dd7-4049-9513-55d519d0230d","Type":"ContainerStarted","Data":"a00c5e682a55f6aedcab457132941b98de92e997010381a4038ceeadee3860aa"} Oct 31 01:06:59.564534 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:59.564365 4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" event={"ID":"d8296280-5dd7-4049-9513-55d519d0230d","Type":"ContainerStarted","Data":"7c91258422a875613d05f12cb15318e52f6dcb32a3b20e4f5ef14f07fff00672"} Oct 31 01:06:59.564534 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:59.564464 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:06:59.582678 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:06:59.582604 4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" podStartSLOduration=1.582591431 podStartE2EDuration="1.582591431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:59.581312829 +0000 UTC m=+2015.610493219" watchObservedRunningTime="2026-10-31 01:06:59.582591431 +0000 UTC m=+2015.611771820" Oct 31 01:07:00.720707 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:00.720678 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-87554bc7c-dpfbs_dd367377-297c-4664-8df8-270c8f55d3b9/console/0.log" Oct 31 01:07:00.753458 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:00.753425 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-qkcwd_95645a0e-1b05-4c37-b42b-b025b3a31ff6/download-server/0.log" Oct 31 01:07:02.568098 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:02.568066 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-72zjc_330ec43b-421e-42ee-b61e-e26743f9fbb7/dns/0.log" Oct 31 01:07:02.600167 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:02.600141 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-72zjc_330ec43b-421e-42ee-b61e-e26743f9fbb7/kube-rbac-proxy/0.log" Oct 31 01:07:02.765571 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:02.765543 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9fz4x_993fd395-dba5-4c79-99d3-72792b8c273c/dns-node-resolver/0.log" Oct 31 01:07:03.484290 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:03.484252 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6bb5c46fb7-x4wbn_753e3824-d0f1-4d84-b2b2-a951de6375f9/registry/0.log" Oct 31 01:07:03.560238 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:03.560210 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n58v5_91f27c45-482c-4a05-998e-18b57ebfa499/node-ca/0.log" Oct 31 01:07:05.578702 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:05.578671 4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bd47j/perf-node-gather-daemonset-drhn8" Oct 31 01:07:05.960297 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:05.960187 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-nnw2v_18daa545-6a52-4ba9-af4b-b60b4ac1f98c/discovery/0.log" Oct 31 01:07:06.019315 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:06.019281 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b544748f5-zkdsq_d7213fb5-28f3-41c1-bd33-4e922f88120d/kube-auth-proxy/0.log" Oct 31 01:07:06.046825 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:06.046788 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-457nv_23d1c38d-9d1d-4674-82cc-38782f2a4f39/istio-proxy/0.log" Oct 31 01:07:06.108777 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:06.108742 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-8wb6m_a11bc6fe-fa8b-425e-b6df-007d3fc6aa07/istio-proxy/0.log" Oct 31 01:07:06.164251 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:06.164218 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-ggdf4_1a6c37d5-3025-470f-94c0-ee1502fc7a76/istio-proxy/0.log" Oct 31 01:07:08.069762 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:08.069735 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vdxdv_9f037fa6-1d56-42b5-922e-007f1af8552c/serve-healthcheck-canary/0.log" Oct 31 01:07:08.793338 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:08.793308 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pgt89_bbc352c4-a2e0-496b-8a9e-56b10e016a74/kube-rbac-proxy/0.log" Oct 31 01:07:08.815932 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:08.815907 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pgt89_bbc352c4-a2e0-496b-8a9e-56b10e016a74/exporter/0.log" Oct 31 01:07:08.838998 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:08.838970 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pgt89_bbc352c4-a2e0-496b-8a9e-56b10e016a74/extractor/0.log" Oct 31 01:07:08.943358 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:08.943324 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-fg9fp-xtjlp_623173be-674f-4b4c-8920-d8b1f213666b/insights-gathering/0.log" Oct 31 01:07:10.413931 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:10.413890 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-nnw2v_18daa545-6a52-4ba9-af4b-b60b4ac1f98c/discovery/0.log" Oct 31 01:07:10.438754 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:10.438720 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b544748f5-zkdsq_d7213fb5-28f3-41c1-bd33-4e922f88120d/kube-auth-proxy/0.log" Oct 31 01:07:10.457098 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:10.457070 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-457nv_23d1c38d-9d1d-4674-82cc-38782f2a4f39/istio-proxy/0.log" Oct 31 01:07:10.495487 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:10.495457 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-8wb6m_a11bc6fe-fa8b-425e-b6df-007d3fc6aa07/istio-proxy/0.log" Oct 31 01:07:10.538847 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:10.538817 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-ggdf4_1a6c37d5-3025-470f-94c0-ee1502fc7a76/istio-proxy/0.log" Oct 31 01:07:11.767614 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:11.767581 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-54459f848c-m9zdn_e9cd99cf-c934-4418-b568-e254d3461a7f/manager/0.log" Oct 31 01:07:11.790245 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:11.790223 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-54459f848c-m9zdn_e9cd99cf-c934-4418-b568-e254d3461a7f/copy-manifests/0.log" Oct 31 01:07:12.548123 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:12.548091 4519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65kvb_3346ce41-dd23-4e63-8944-43f2d7d4aaec/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:13.662837 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:13.662791 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8kw7h_43e494ff-c9a1-4707-8f27-efd46b218d97/manager/0.log" Oct 31 01:07:14.054529 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:14.054492 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-hzz82_3147d13d-fa94-4a4c-a468-b7c9e275ef43/manager/0.log" Oct 31 01:07:14.144379 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:14.144345 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-nnw2v_18daa545-6a52-4ba9-af4b-b60b4ac1f98c/discovery/0.log" Oct 31 01:07:14.207741 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:14.207713 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b544748f5-zkdsq_d7213fb5-28f3-41c1-bd33-4e922f88120d/kube-auth-proxy/0.log" Oct 31 01:07:14.225477 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:14.225450 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-457nv_23d1c38d-9d1d-4674-82cc-38782f2a4f39/istio-proxy/0.log" Oct 31 01:07:14.264955 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:14.264927 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-8wb6m_a11bc6fe-fa8b-425e-b6df-007d3fc6aa07/istio-proxy/0.log" Oct 31 01:07:14.306553 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:14.306469 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-ggdf4_1a6c37d5-3025-470f-94c0-ee1502fc7a76/istio-proxy/0.log" Oct 31 01:07:14.642400 ip-10-0-143-188 kubenswrapper[4519]: I1031 01:07:14.642322 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-hkvls_c0bb009c-f110-4275-8030-46207de5b3e2/proxy/0.log"